Pandas Datareader with Morningstar


Pandas Datareader with Morningstar



I'm web scraping with pandas and using morningstar's API. Every now and then I scrape a website with a ticker that morningstar struggles with and crashes python.



I've included relevant lines below and it almost always work and I don't think my code is causing the errors but I can't get around pandas freezing up on me. Date is a variable here that is in the correct format for DataReader, it doesn't raise problems.


import pandas as pd
import pandas_datareader.data as web

df = web.DataReader(ticker, "morningstar", date)



If a ticker is getting stuck, after a couple minutes pandas will print (on it's own, not my print statement) the phrase "adding (insert ticker here) to retry list". Shortly after the message pops up I get a pop up window saying "Python quit unexpectedly".



I've tried to wrap the datareader in a while loop with a timer to stop it and move on to the next ticker but it didn't work. I'm assuming this is because of control flow.
I've also tried to use the interruptingcow module in hopes it could help but got the same result as a regular timer.



Is there a way to get around this? It crashes my script every time.
Some tickers I've come across that are a pain are 'BBXTB', 'CRXPF', 'IMMVD', and 'FFRMF'




1 Answer
1



I still do not know exactly why the tickers would fail and am investigating. however, I was able to allow the failure to not crash the scripts by directly modifying pandas_datareadermstardaily.py.


pandas_datareadermstardaily.py





My version is 0.6.0. The instructions are for this version only.


0.6.0





lines 134 - 138 looks like this:


if len(failed) > 0 and self.retry_count > 0:
# TODO: This appears to do nothing since
# TODO: successful symbols are not added to
self._dl_mult_symbols(symbols=failed)
self.retry_count -= 1



the counter is in the wrong place and needs to be swapped with the above line, as such:


if len(failed) > 0 and self.retry_count > 0:
# TODO: This appears to do nothing since
# TODO: successful symbols are not added to
self.retry_count -= 1
self._dl_mult_symbols(symbols=failed)



this should allow the failed call to exit gracefully without causing a stack overflow.



incidentally, https://github.com/pydata/pandas-datareader/blob/master/pandas_datareader/mstar/daily.py shows that @dtemkin has directly fixed the current build of pandas_datareader so this may be released/resolved on next release.





You can find this file by locating where python stores its libraries. I installed anaconda through Visual Studio 2017, and the location of the file is at: C:Program Files (x86)Microsoft Visual StudioSharedAnaconda3_64libsite-packagespandas_datareadermstardaily.py


C:Program Files (x86)Microsoft Visual StudioSharedAnaconda3_64libsite-packagespandas_datareadermstardaily.py






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

api-platform.com Unable to generate an IRI for the item of type

api-platform.com Unable to generate an IRI for the item of type I'm trying to setup my first api-platform instance and have run into some presumably noob questions around IDE. What i'm trying to do is setting up an register and reset entity in order to deal with registrations and password reminders from an app. And then have the entity's passed to a custom controller to do some checks, balances and mails afterwards (haven’t gotten around to that yet). But I keep running into this error when posting to my custom entities: { "@context": "/api/contexts/Error", "@type": "hydra:Error", "hydra:title": "An error occurred", "hydra:description": "Unable to generate an IRI for the item of type "AppXXXUserBundleEntityRegister"", "trace": [ { "namespace": "", "short_class": "", "c
Read more

How to set up datasource with Spring for HikariCP?

How to set up datasource with Spring for HikariCP? Hi I'm trying to use HikariCP with Spring for connection pool. I'm using jdbcTempLate and JdbcdaoSupport. This is my spring configuration file for datasource: <bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource"> <property name="dataSourceClassName" value="oracle.jdbc.driver.OracleDriver"/> <property name="dataSource.url" value="jdbc:oracle:thin:@localhost:1521:XE"/> <property name="dataSource.user" value="username"/> <property name="dataSource.password" value="password"/> </bean> But unfortunately the following error message is generating: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource'
Read more

Display dokan vendor name on Woocommerce single product pages

Display dokan vendor name on Woocommerce single product pages With woocommerce I am using Dokan plugin and I am trying to display the vendor name, rating and vendor location on single product pages. I tried this code to display vendor name but no luck: //show store name add_action( 'woocommerce_single_product_summary', 'show_store_name', 20 ); function show_store_name() { printf( '<b>Seller Name:</b> <a href="%s">%s</a>', dokan_get_store_url( $author->ID ), $store_info['store_name'] ); } Any help is appreciated. 1 Answer 1 I don't use dokan plugin. Here is the way to get the (post) author ID and to make your code work: add_action( 'woocommerce_single_product_summary', 'display_author_name', 20 ); function display_author_name() { // Get the author ID (the vendor ID) $vendor_id = get_post_field( &#
Read more