New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
getSymbols.yahoo does not follow redirect to HTTPS #149
Comments
I can confirm this problem. |
Same problem. It seams that yahoo changed this. |
This appears to be caused by a redirection from HTTP to HTTPS. I'm running R-3.3.3 with libcurl capabilities, so this still works, but with a warning. R> getSymbols("SPY", verbose = TRUE)
downloading SPY .....
trying URL 'http://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=2007&d=3&e=18&f=2017&g=d&q=q&y=0&z=SPY&x=.csv'
Warning in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
"internal" method cannot handle https redirection to: 'https://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=2007&d=3&e=18&f=2017&g=d&q=q&y=0&z=SPY&x=.csv'
switching to method = "libcurl" because of redirection to https
trying URL 'http://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=2007&d=3&e=18&f=2017&g=d&q=q&y=0&z=SPY&x=.csv'
downloaded 187 KB
done.
[1] "SPY"
R> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quantmod_0.4-7 TTR_0.23-1 xts_0.9-7 zoo_1.7-13
loaded via a namespace (and not attached):
[1] grid_3.3.3 lattice_0.20-34
R> capabilities('libcurl')
libcurl
TRUE |
Well, I have libcurl so it looks something else is wrong: > capabilities('libcurl')
libcurl
TRUE
> getSymbols("SPY", verbose=TRUE)
downloading SPY .....
trying URL 'http://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=2007&d=3&e=18&f=2017&g=d&q=q&y=0&z=SPY&x=.csv'
Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
cannot open URL 'http://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=2007&d=3&e=18&f=2017&g=d&q=q&y=0&z=SPY&x=.csv' |
@SamoPP But you're using an older version of R (3.3.1). That version might not automatically attempt to switch to libcurl in order to follow a redirect. |
Well, what is funny is this... If I run the above in terminal I get the error. If I run the above in StatET in Eclipse I get the error. If I run the same thing in RStudio it works... I hate changing R versions since doing that every now and then breaks a whole lot of things... |
I'm using R 3.3.2 it won't work in R, but it works in RStudio. |
Confirmed that this issue affects R 3.3.1. It works from RStudio, but fails when run from RScript. |
Yahoo Finance is now redirecting HTTP requests to HTTPS. Some users' R installations are not able to follow this redirect, so they error. Use HTTPS directly for Yahoo Finance and Yahoo Finance Japan. Fixes #149.
Just pushed a patch to branch 149_yahoo_https. Works for me on Ubuntu. Would be great to have confirmation it works on Windows and MacOS. |
Looks good for getSymbols(). Thanks Now just adjustOHLC() is not wrking... :) $ R
R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library(quantmod)
Loading required package: xts
Loading required package: zoo
Attaching package: ‘zoo’
The following objects are masked from ‘package:base’:
as.Date, as.Date.numeric
Loading required package: TTR
Version 0.4-0 included new data defaults. See ?getSymbols.
> getSymbols("SPY")
As of 0.4-0, ‘getSymbols’ uses env=parent.frame() and
auto.assign=TRUE by default.
This behavior will be phased out in 0.5-0 when the call will
default to use auto.assign=FALSE. getOption("getSymbols.env") and
getOptions("getSymbols.auto.assign") are now checked for alternate defaults
This message is shown once per session and may be disabled by setting
options("getSymbols.warning4.0"=FALSE). See ?getSymbols for more details.
[1] "SPY"
> SPY <- adjustOHLC(SPY)
Error in download.file(paste(yahoo.URL, Symbol.name, "&a=", from.m, "&b=", :
cannot open URL 'http://ichart.finance.yahoo.com/table.csv?s=SPY&a=0&b=01&c=1900&d=3&e=18&f=2017&g=v&ignore=.csv'
> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quantmod_0.4-7 TTR_0.23-1 xts_0.10-0 zoo_1.8-0
loaded via a namespace (and not attached):
[1] grid_3.3.1 lattice_0.20-34 |
Doesn't quite seem to work for Windows (but I'm not an R guru so I could easily be mistaken). I'm running R 3.1.2. I tried the following code: symbol <- "SPY"
from <- "2016-04-17"
to <- "2017-04-17"
yahoo.URL <- "https://ichart.finance.yahoo.com/table.csv?"
from.y <- as.numeric(strsplit(as.character(as.Date(from, origin = "1970-01-01")), "-", )[[1]][1])
from.m <- as.numeric(strsplit(as.character(as.Date(from, origin = "1970-01-01")), "-", )[[1]][2]) - 1
from.d <- as.numeric(strsplit(as.character(as.Date(from, origin = "1970-01-01")), "-", )[[1]][3])
to.y <- as.numeric(strsplit(as.character(as.Date(to, origin = "1970-01-01")), "-", )[[1]][1])
to.m <- as.numeric(strsplit(as.character(as.Date(to, origin = "1970-01-01")), "-", )[[1]][2]) - 1
to.d <- as.numeric(strsplit(as.character(as.Date(to, origin = "1970-01-01")), "-", )[[1]][3])
verbose <- TRUE
tmp <- tempfile()
download.file(paste(yahoo.URL, "s=", symbol, "&a=", from.m, "&b=", sprintf("%.2d", from.d), "&c=", from.y, "&d=", to.m, "&e=", sprintf("%.2d", to.d), "&f=", to.y, "&g=d&q=q&y=0", "&z=", symbol, "&x=.csv", sep = ""), destfile = tmp, quiet = !verbose)
fr <- read.csv(tmp)
unlink(tmp) And got the following error: That was the same code that had worked prior to yesterday with the exception of the change in the yahoo url. Best regards. |
@SamoPP Good catch. That's @rmicalet R-3.1.2 is 2.5 years old and HTTPS wasn't supported and used as a default on all platforms until R-3.3.0. So that's the likely cause of your issue, not the code in |
Ok, updated to R 3.3.3 and everything seems to work. Thank you! |
Just pulled 149_yahoo_https branch, built it and tested on ubuntu. Works for me now. Thank you! |
I still have Problems in R 3.1.1 (I know it's old but it's the only version that runs in raspberry pi) with the new version quantmod 0.4-8. Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, : |
@actuariat As I said earlier, HTTPS wasn't supported and used as a default on all platforms until R-3.3.0. I just pushed a branch (your username) that replaces Note to self: |
I see this bug has been closed (and |
@my-R-help It works for me and others, so you need to provide more information than "still doesn't work" in order to help me debug. At minimum, please provide the error and the output from |
Interesting, I just realized that it works like this:
But it does NOT work like this:
|
Or you can just use |
Great, that makes total sense. Many thanks for all your great work on quantmod! |
I had this problem too, thanks Josh for the fix. Updating to quantmod 0.4-8 fixed for me on Os X 10.10.5 and R 3.3.0. In case this is useful to anyone, I added the following line before calling getSymbols.yahoo as a workaround before Josh pushed his fix: |
Has this bug reappeared in R 3.4.0?
Matrix products: default locale: attached base packages: other attached packages: loaded via a namespace (and not attached): |
No, this is a different issue. A "502 Bad Gatway" reponse is different from a "30X redirect" response. Not being able to follow the redirect was the issue here. |
Hi, I am also seeing this error - HTTP status was '502 Bad Gateway'. Here is the output of my sessionInfo() command :
Matrix products: default locale: attached base packages: other attached packages: loaded via a namespace (and not attached): |
getSymbols seems to fail if the source is Yahoo; looks like Yahoo changed the link for historical data, so https://ichart.finance.yahoo.com/table.csv no longer works. Anybody has a clue how to fix this? |
It looks like Yahoo has just stopped that service, see this: https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/5 |
Yahoo is just sucking, you can force it to use Google like so: |
Hi guys, I am experiencing error in getQuote. Has anyone seen this recently ?
|
at 4pm yesterday right when the markets closed it seems yahoo dropped the whole csv quote system, they made their historical data harder to scrape a few months ago so this just seems to be more going in the way of stopping scraping of their data. |
#' Yahoo GetQuote |
installing quantmod_0.4-11 gihub gets a working version of getSymbols |
I also get error: from <- c("USD", "USD", "EUR", "GBP", "EUR", "EUR") quantmod v0.4.11 |
Hi, It was working yesterday, today morning IST , getSymbols.yahoo broke
But this seems to work:
but getSymbols google doesn't download information about indian stocks like :
but this used to work till yesterday:
how to get it to work ?? |
Broken even with latest quantmod from github.sessionInfo() Matrix products: default locale: attached base packages: other attached packages: loaded via a namespace (and not attached): getSymbols("SPY", verbose = TRUE) 100 125 100 125 0 0 1216 0 --:--:-- --:--:-- --:--:-- 1275 |
@tvkurien Your issue is not a HTTP -> HTTPS redirection issue (i.e. is not related to this issue), and has likely been fixed in the |
I'm locking this issue to avoid more comments about unrelated issues. |
It might be that Yahoo again changed something since today download is not working for me any more... I have tried the below many times on many different symbols but no luck...
The text was updated successfully, but these errors were encountered: