Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

getOptionChain() working discretely "Error in open.connection(con, "rb") : HTTP error 500." #299

Closed
jrburl opened this issue Feb 20, 2020 · 11 comments
Closed

getOptionChain() working discretely "Error in open.connection(con, "rb") : HTTP error 500." #299

jrburl opened this issue Feb 20, 2020 · 11 comments
Milestone
Release 0.4.18

Comments

@jrburl
Copy link
Contributor

jrburl commented Feb 20, 2020

Description

thanks for your work
Unable to pull option chains for a few high profile companies. With the same code, am able to pull for other companies. Does work for 'F', doesn't work for 'MSFT', 'AAPL'.

Expected behavior

Expecting a response.

Minimal, reproducible example

library(ggplot2)
library(reshape2)
library(lubridate)
library(xlsx)
library(quantmod)
library(readxl)
library(purrr)
library(dplyr)
library(plotly)
library(BatchGetSymbols)
library(derivmkts)

test_ticker <- 'AAPL'
test_total = data.frame()
spot_check <- 
  tryCatch(
  test <- as.matrix(quantmod::getOptionChain(test_ticker,Exp = NULL)),
      error=function(e) e)
  mylist = do.call(rbind,do.call(rbind,test)) %>%
    tibble::rownames_to_column(., "Name")
  
  test_total <- rbind(test_total,mylist)

  View(test_total)
@joshuaulrich
Copy link
Owner

Thanks for the report. I cannot replicate with a simple, minimal example:

x <- quantmod::getOptionChain("AAPL", Exp = NULL)
length(x)
# [1] 18
str(x[1:2])
# List of 2
#  $ Feb.21.2020:List of 2
#   ..$ calls:'data.frame':	78 obs. of  7 variables:
#   .. ..$ Strike: num [1:78] 100 105 110 120 130 135 140 150 155 160 ...
#   .. ..$ Last  : num [1:78] 224 219 214 205 194 ...
#   .. ..$ Chg   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:78] 2 NA 5 NA NA 3 7 180 NA 31 ...
#   .. ..$ OI    : int [1:78] 0 0 0 NA 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame':	74 obs. of  7 variables:
#   .. ..$ Strike: num [1:74] 100 105 110 115 120 125 130 135 140 145 ...
#   .. ..$ Last  : num [1:74] 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.03 0.03 0.04 ...
#   .. ..$ Chg   : int [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : num [1:74] 0.01 0.04 0.01 0 0 0.08 0.08 0.01 0 0 ...
#   .. ..$ Vol   : int [1:74] 8 96 124 270 32 25 15 45 35 1 ...
#   .. ..$ OI    : int [1:74] 1724 0 193 0 0 85 89 110 0 0 ...
#  $ Feb.28.2020:List of 2
#   ..$ calls:'data.frame':	60 obs. of  7 variables:
#   .. ..$ Strike: num [1:60] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:60] 127 112 114 103 104 ...
#   .. ..$ Chg   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:60] 15 24 45 10 67 28 1 1 2 15 ...
#   .. ..$ OI    : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame':	58 obs. of  7 variables:
#   .. ..$ Strike: num [1:58] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:58] 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.02 0.01 0.02 ...
#   .. ..$ Chg   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:58] 0 0 0 0 0.01 0 0 0.03 0 0 ...
#   .. ..$ Ask   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:58] 9 168 602 402 402 49 29 49 7 113 ...
#   .. ..$ OI    : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

A HTTP 500 error is a generic error that suggests something went wrong with the Yahoo Finance server(s). I suspect your problem will be resolved once your request goes to a functioning Yahoo Finance server.

So I'm going to close this because I don't think there's anything I can do to help resolve your issue. Please feel free to re-open it if you think I've missed something.

Also, I'd appreciate if your minimal reproducible example were more minimal. I hope you would agree that it's not minimal to attach 11 packages (and however many their Depends also attach) that require installation of nearly 100 packages. It probably didn't matter in this case, but interactions between attached packages can potentially cause issues that are very hard to track down.

@jrburl
Copy link
Contributor Author

jrburl commented Feb 20, 2020 via email

@joshuaulrich
Copy link
Owner

Here's another example of an error caused by the Yahoo Finance server response.

I ran this code 2 days ago and got an error:

R> x <- quantmod::getOptionChain("MSFT", Exp = NULL)
Error in data.frame(Strike = strike, Last = lastprice, Chg = change, Bid = bid,  : 
  object 'volume' not found

I ran it again just now and it worked. It might have something to do with the weekly options. We might be able to address that in quantmod, if we can find the root cause of the error. I'd appreciate it if you're able to investigate. If not, no worries.

@jrburl
Copy link
Contributor Author

jrburl commented Mar 1, 2020

@joshuaulrich
quantmod::getOptionChain('PG', NULL) is failing for the same reasons. I went and checked out the option chain on Yahoo Finance and if you check out the 4/9/2020 expiry it's got near zilch for volume. No volume on calls, 1 on puts. I believe this is the culprit.

quantmod::getOptionChain('PG', Exp = "2020-04-09") fails w/ the same error & adds to this idea.

@jrburl
Copy link
Contributor Author

jrburl commented Mar 1, 2020 •
edited

Have submitted a fork request. I think this is what you're looking for after your NewToOld function:

                        Vol= if("volume" %in% names(x))
                        {
                          volume
                        }
                        else
                        {
                          NA
                        }

Also. The same needs to be done for open interest. I've seen that fail for the same reason as well.

@joshuaulrich
Copy link
Owner

I see your patch, thanks! It looks ready to merge to me, but you need to click the [Create Pull Request] button for me to be able to merge. Thank you very much for taking the initiative to fix that issue!

jrburl referenced this issue Mar 13, 2020
@jrburl
Update getOptionChain to handle empty volume or open interest
bb4be7c 
Have seen getOptionChain fail on several occasions due to 
"Error in data.frame(Strike = strike, Last = lastprice, Chg = change, Bid = bid,  :  object 'volume' not found". (or open interest). Proposing an if else statement to check if "volume" and "openinterest" are in the table column names. If in, no changes, if out, populate with NA.
@joshuaulrich
Copy link
Owner

As @jrburl mentioned in the commit comment, this error occurred again. Thankfully, I can replicate, so I'm reopening.

Here's some relevant information from stepping through the debug call that fails.

debugonce(quantmod:::getOptionChain.yahoo)
oc <- quantmod::getOptionChain("FB", Exp = "2020-04-09")
...
# Browse[2]> all.expiries.posix
 [1] "2020-03-13 UTC" "2020-03-20 UTC" "2020-03-27 UTC" "2020-04-03 UTC"
 [5] "2020-04-09 UTC" "2020-04-17 UTC" "2020-04-24 UTC" "2020-05-01 UTC"
 [9] "2020-05-15 UTC" "2020-06-19 UTC" "2020-09-18 UTC" "2020-11-20 UTC"
[13] "2021-01-15 UTC" "2021-03-19 UTC" "2021-06-18 UTC" "2022-01-21 UTC"
[17] "2026-02-21 UTC"
...
Browse[2]> 
debug: return(getOptionChain.yahoo(Symbols, expiry.subset, .expiry.known = TRUE))
Browse[2]> expiry.subset
[1] 1586390400
Browse[2]> .POSIXct(expiry.subset, tz = "UTC")
[1] "2020-04-09 UTC"
Browse[2]> 
Error in open.connection(con, "rb") : HTTP error 500.

So here's some relevant information from stepping through a debugging call to getOptionChain.yahoo()

quantmod:::getOptionChain.yahoo("FB", 1586390400, .expiry.known = TRUE)
# Error in open.connection(con, "rb") : HTTP error 500.
debugonce(quantmod:::getOptionChain.yahoo)
...
Browse[2]> 
# debug: tbl <- jsonlite::fromJSON(urlExp)
Browse[2]> urlExp
# [1] "https://query2.finance.yahoo.com/v7/finance/options/FB?&date=1586390400"
Browse[2]> 
# Error in open.connection(con, "rb") : HTTP error 500.

@joshuaulrich joshuaulrich reopened this Mar 13, 2020
@joshuaulrich
Copy link
Owner

Something is odd with the site. I'm able to select the 2020-04-09 expiry from the drop-down, but all the contracts listed in the output do not change. The values are for whatever expiry was last selected. In the screenshot below, that's 2020-04-03.

Screenshot from 2020-03-12 21-05-50

@jrburl
Copy link
Contributor Author

jrburl commented Mar 28, 2020 •
edited

Good evening. The flavor of the day is GOOG200501. Appears to be on yahoo's end.
Screen Shot 2020-03-27 at 8 32 19 PM
Screen Shot 2020-03-27 at 8 53 18 PM
Screen Shot 2020-03-27 at 8 33 21 PM

@joshuaulrich
Copy link
Owner

Hey @jrburl, what do you think of the patch I just pushed?

Here's an example of the output when Exp = NULL and there are expiry dates that do not have data. In this case, oc will not have list elements named "Jun.15.2020", "Jun.17.2020", "Jun.26.2020", or "Mar.18.2022".

oc <- quantmod::getOptionChain("SPY", NULL)
Warning: no data for 'SPY' expiry 2020-06-15, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-17, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-26, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2022-03-18, omitting
	(server response: HTTP error 500.)

@ekuutan
Copy link

ekuutan commented May 16, 2020

It stopped working altogether for me "Error in open.connection(con, "rb") : HTTP error 500"

@joshuaulrich joshuaulrich added this to the Release 0.4.18 milestone Nov 28, 2020
@joshuaulrich joshuaulrich mentioned this issue Nov 29, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Release 0.4.18
Development

No branches or pull requests
3 participants
@joshuaulrich @ekuutan @jrburl