Yahoo Finance returns 502 Bad Gateway #157
Comments
|
I also noticed this bug today, but how do we deal with the crumb parameter? It's required since after https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492379145&period2=1494971145&interval=1d&events=history (i.e. without the crumb param) I get Any ideas here? |
|
You need to establish a session with the Yahoo Finance server in order to get the crumb. That will probably require using curl (via either the RCurl or curl R package). |
|
Can someone post an example of how to use RCurl + getSymbols etc? |
Yahoo changed their URL scheme to require an established session with cookies. The 'curl' package handles all the details. First establish a connection to the main site to get a cookie, then use that to get a crumb, and use the crumb in the download URL. Note that times also changed to POSIX/UNIX timestamps. No additional changes were needed for getDividends(). getSplits() was updated to read the new CSV format and parse the new split format. getSymbols.yahoo() needed to re-order the columns, because Volume and Adj.Close were swapped. It also appears the Open, High, and Low columns are already adjusted for splits, but not dividends. Un-adjust them for splits so the result is consistent with prior behavior. See #157.
Keep a reference to the curl handle in quantmod's private environment. This avoids having to re-establish a connection for every query. The .getHandle() function will establish a session the first time it's called. Then it will simply return the handle for subsequent calls. Note that the curl package author recommends to not re-use curl handles. I think it's okay in this case because we're always referencing the same site. If we need to use curl for multiple sites, we should establish handles for each, individually. See #157.
Setting from="1900-01-01" resulted in a bad request. Pre-1950 should be adequate for the majority of companies. See #157.
|
I'm sorry, but I'm confused as to how to use the updates. How do I update the functions with the changes you just made? |
|
@lompoc42 one of these: remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")Or wait until it's merged to master and use: remotes::install_github("joshuaulrich/quantmod")
# or
devtools::install_github("joshuaulrich/quantmod") |
|
THANK YOU SO MUCH!!!!!!!!!!! |
|
Getting an error with the update: |
|
@lompoc42 Please provide the command that causes that error. It's nearly impossible to help without it. |
|
In RStudio |
|
Hey Joshua, thanks for giving us quantmod, and thanks for the quick fix. After installing the update I can download FB and APPL historical prices with, e.g., loadSymbols("FB"). I get the same error as lompoc42 when trying ^GSPC, ^GDAXI, SIE.DE, or LQD. It seems the query string varies with the ticker (just a shot in the dark, no time to check). Cheers, |
|
Yeah, it works for some but not for others. Got the same error also with tickers: PRWCX, VBTLX |
The downloaded file may have "null" for some observations. Use the na.strings argument to convert them to NA. See #157.
|
The issue with the symbols listed should now be resolved. Thanks for the testing and feedback! |
|
The '502 Bad Gateway' error is back. I didn't change anything which was working before and now no longer working. I tried re-installing the package you referenced earlier
and am getting the following error at the end
|
|
Not sure what's happened, but it's not working anymore |
|
It is now working for me. Thank you for your help. However, it looks like it is only returning data going back to 2014. Also, the finance.yahoo.com site seems to be having issues returning older data. |
@lompoc42 it works fine for me from a fresh install on another machine. I assume you're running from RStudio; have you tried restarting RStudio? Unload/reload doesn't always work well. If that doesn't work, see if it works if you run from the GUI that comes with R and/or the terminal.
@bdfden I just pulled data from 2007. If you're only getting data back to 2014 for a specific symbol, and you get the same behavior for the same instrument on the website, then there's no issue with quantmod. |
|
Agreed - this issue seems to be with Yahoo. Thanks again. |
|
Just want to thank you for the great work on the quantmod package. I spent a few weeks developing a shiny app that could mimic yahoo charts in April 2017...my goal was to showcase what is now possible using the shiny framework & R's immense library of packages. I'm using the highcharts R package which I also highly recommend for charting. Here is a screen shot.
When I noticed the 502 Bad Gateway yesterday I feared the worse. But I've implemented your update and the app is working as per usual. So thanks very much for all your great work! |
|
Woah. Your shiny app looks much better than mine!
… On May 17, 2017, at 7:46 PM, Ashley Joyce ***@***.***> wrote:
@joshuaulrich
Just want to thank you for the great work on the quantmod package. I spent a few weeks developing a shiny app that could mimic yahoo charts in April 2017...my goal was to showcase what is now possible using the shiny framework & R's immense library of packages. I'm using the highcharts R package which I also highly recommend for charting.
Here is a screen shot.
When I noticed the 502 Bad Gateway yesterday I feared the worse. But I've implemented your update and the app is working as per usual.
So thanks very much for all your great work!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
if you happen to encounter the following error while installing under OS X.... the fix is to to |
|
@mcohen01 See the second-to-last bullet in the Installation section of quantmod's README: |
|
Thanks for Quantmod Joshua, I'm very new to R and I have been using Quantmod for a couple of months and its been very useful. However, I ran into the same problem other users have talked about. Just wanted to ask you if you are planning to release a new version or should I stick to the fixes you listed before. I tried to run them but I'm getting errors. Thanks again! |
|
Thank you for your help. Should I make any change in Ad(get(symbol[i]]) to get prices from multiple tickers? |
|
Thank you so much, it works now after installed devtools provided by Joshua. I can check stock data as usual now. But it looks like yahoo finance server miss a lot of daily information in some TW STOCKs. |
The attempt to establish a session could result in a curl handle that does not have a cookie, in which case the user will not be able to import data. Establish a connection and check if it resulted in a cookie. Try again, up to 5 times, if it does not. Throw an error if the attempt fails 5 times. Hopefully most users will not receive errors because of this, combined with the second attempt to import data if the first attempt fails, See #157.
|
@joshuaulrich I have ran your updates and I am still getting the 502 Bad Gateway issue...is there anything you recommend (with an example) for how I can get it to work. Thank you. |
|
@mlb92 You have not run the commands to install the patch. The commands you used will only work once the patch is merged, which it has not been. You need to run these commands: remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") |
|
@joshuaulrich I am very new to programming, could you explain what you mean by "You have not run the commands to install the patch."? I have run these lines so far. Thank you for your help: install.packages("remotes") After running these, I get the following errors
|
|
@mlb92 Then you should just wait until this is fixed. I told you to run |
|
I ran devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") and ran getSymbols("AAPL"). I get the follwoing: Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, : I am using windows and latest version - R 3.4.0 (also got the same error previously with R 3.3.2) Is yahoo! down? I tried opening https://chart.finance.yahoo.com/ and I get an error message from Yahoo! saying "Will be right back.." |
|
@umeshnathani I think you might need to restart your R session after installing the branch (157_yahoo_502) or just remove quantmod package you already have and then run: I just tested AAPL and it worked fine, here are the results and session info: |
|
@joshuaulrich The current build of getSymbols.yahoo on github is removing downloaded rows which have a date but only NA prices and volumes. An example is VDIGX for 2016-06-29. This seems to be a change from last week's version which didn't remove these rows. I had been using na.approx(..., maxgap = 2) to deal with a small number of rows missing in the download and that worked for my purposes. Could removing missing rows be made an option with a warning if rows were missing rather than the fixed behavior? |
Removing missing values assumes that's what the user prefers. The object contains the na.omit attribute, so they could call na.restore() to put the missing values back. However, that's more work and makes it easier to ignore the fact that the data are suspect. Update the warning to explain why the missing values might be a problem and suggest possible solutions. See #157.
This allows the .yahooURL() function to be used in getDividends and getSplits(), which reduces duplicated code. Also use the .dateToUNIX() function to calculate the from and to POSIX dates in both functions. See #157.
Similar to the first call to getSymbols, the first call to getSymbols.yahoo should throw a warning alerting users to the changes in the Yahoo Finance data. Include more details in ?getSymbols.yahoo. Also update the message printed after the first call to getSymbols(). See #157.
|
The Thanks for everyone's feedback, testing, and support! |
|
Hi Mr Joshualrich, I´m Lois. Since past may 12 of 2017, WannaCry event, I can´t see this chart (Nike Inc. stok) in my browser: http://ichart.finance.yahoo.com/z?s=NKE&t=6m&q=c&l=on&z=m&p=e50,e20&a=v,r14 Dear Joshualrich, ¿Is there any way to fix this problem? You said that was a change of URL, this is, ¨https¨ instead of ¨http¨ Thanks for your time. Bye for now |
Use temporary fix from 'quantmod' pkg, hopefully to be included by default soon. Ref: joshuaulrich/quantmod#157
Setting from="1949-01-01" in 05e87ea worked because it avoided date-to-POSIX conversion error that was fixed in 97e4df6. The issue is that the POSIXct value for 1900-01-01 is just slightly less than the 32-bit integer minimum. Return the defaults to their original value, since this isn't a problem for the Yahoo Finance servers. See #157.
|
Yahoo finance API is not available anymore. I have moved to MarketXLS after this change, much more reliable data. |
|
@Harrymon12 It simply isn't true that the Yahoo Finance API is not available. I just ran this code: R> quantmod::getSymbols("SPY", src = "yahoo")
[1] "SPY"
R> head(SPY)
SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.Adjusted
2007-01-03 176.779 177.537 174.691 141.37 94807600 113.7571
2007-01-04 175.512 176.531 174.741 141.67 69620600 113.9985
2007-01-05 175.636 175.723 174.455 140.54 76645300 113.0893
2007-01-08 175.002 175.735 174.294 141.19 71655000 113.6122
2007-01-09 175.611 175.971 174.480 141.07 75680100 113.5157
2007-01-10 174.704 175.934 174.356 141.54 72428000 113.8939
R> packageVersion("quantmod")
[1] '0.4.10' |
|
@joshuaulrich Oh I see. It came back. |
|
It looks like same underlying data source is used for both the query1 and query2 APIs. E.g. the WBA issue that @lowcsharp mentioned (#157 (comment)) is an issue in both endpoints |
and others
After switching from HTTP to HTTPS (#149), it looks like Yahoo has now changed the URL.
The "Download Data" link on the historical data page for AAPL is now:
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492379145&period2=1494971145&interval=1d&events=history&crumb=sgdBTt48gg.
The text was updated successfully, but these errors were encountered: