• Play
  • About
  • News
  • Forums
  • Yppedia
  • Help
Welcome Guest   | Login
  Index  | Recent Threads  | Register  | Search  | Help  | RSS feeds  | View Unanswered Threads  
  Search  


Quick Go »
Thread Status: Normal
Total posts in this thread: 10
[Add To My Favorites] [Watch this Thread] [Post new Thread]
Author
Previous Thread This topic has been viewed 1227 times and has 9 replies Next Thread
Pauling

Member's Avatar


Joined: Dec 20, 2005
Posts: 1997
Status: Offline
Monitoring commodity prices over time? Reply to this Post
Reply with Quote

In order to run my shops more effectively, I'd like to get a better feel for how commodity prices vary over time. Years ago, rustycutlass.org offered a way to do this by monitoring tax prices, which you could then graph to see ocean-wide price cycles over the span of years.

Is there a more modern, and still-functioning, site that replaces rustycutlass? Ideally a site that lets you download the raw data?

As of last night, I've begun automated tracking of tax data for my own private consumption, but obviously it will be some time before I start accumulating enough information to show long term trends for the three oceans.
----------------------------------------
Pauling of the Lost Lot, Sagemerald
Officer emeritus and occasional cabin person
Puppetar (and drawing of secret Blackthorpe plans) by the talented Tilinka
[Apr 2, 2012 8:29:19 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
zilenserz

Member's Avatar


Joined: Mar 15, 2007
Posts: 200
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

How're you automating it? I was thinking about doing a manual version for myself, but it'd be really time consuming...
----------------------------------------
Zilenserz on Hunter,
Sailing round in circles sincle April '07.
 
Duckling flag officer chats, "What is this RL"

[Apr 2, 2012 10:17:48 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Pauling

Member's Avatar


Joined: Dec 20, 2005
Posts: 1997
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

 
How're you automating it? I was thinking about doing a manual version for myself, but it'd be really time consuming...


I wrote a quick and dirty python script that parses the tax table with BeautifulSoup, and runs nightly on a web server via a cron job. Then, the results are written as a new line to a CSV file (one new line per day, one file per ocean). For manipulation, I loaded the data into Google Docs / Excel using importData. Of course, I only have 2 days of data at the moment, so it's pretty manageable. :)

It's fine for personal use, but for a web app that others could use, I don't think that reading an entire ginormous CSV file for any of a thousand potential users is the right way to go about it. I lack experience with databases, though. As data continues to accumulate, I'd be open to suggestions, collaborations, or data sharing with more dev-savvy pirates.

(I'm not averse to sharing the code- but I'd prefer to see a single centralized service rather than a dozen web server scripts that not everyone can use)
----------------------------------------
Pauling of the Lost Lot, Sagemerald
Officer emeritus and occasional cabin person
Puppetar (and drawing of secret Blackthorpe plans) by the talented Tilinka
----------------------------------------
[Edit 1 times, last edit by Pauling at Apr 2, 2012 11:55:19 AM]
[Apr 2, 2012 11:51:59 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
SirCarl67

Member's Avatar


Joined: Feb 5, 2006
Posts: 2090
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

I use this.
----------------------------------------
Apollo tells ye "If you're ok for me to do one [bake-off], I'll go with that."
----------------------------------------
[Edit 1 times, last edit by SirCarl67 at Apr 2, 2012 11:58:28 AM]
[Apr 2, 2012 11:57:53 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
wrs1864b

Member's Avatar


Joined: Apr 10, 2006
Posts: 5450
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

 
I use this.

That is the only publicly available source I know of also. I don't find it very useful since you can't change the time scales and a lot of minerals vary over several years.

I have spotty data for Cobalt going back to 2010, but that is only useful enough to show to me that the tax values aren't that useful. For example, tax values on Cobalt depended a great deal on whether I was active in the game, doing trade runs and selling stuff to many people, or I'm away from the game and no one else is performing that function. Tax values are a bad mixture of too many different prices, many of which are easily manipulated by players, either accidentally or on purpose.

Tax values are nice to figure out what the sales tax will be for a product, but it can be dangerous to read too much into them for long term trends. Heck, even trying to use tax info or products can fail since not all tax values for all ingredients are published by OOO. The only way to get the tax values for all commodities is to have a BK holding an island, and then you can look at the fort buy/sell prices. (BKs set the buy price to tax*10*0.90 and the sell to tax*10*1.10.)
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.
[Apr 2, 2012 12:21:39 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Pauling

Member's Avatar


Joined: Dec 20, 2005
Posts: 1997
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

Thanks for the tip! I've been planning to fetch tax data daily, but the right-hand plots on weathered helm seem to feature far more fine grained data (and changes over smaller intervals). Does anyone know what the ideal sampling interval is?

I can keep fetching data for now... tax rates aren't great, but they're better than nothing. (I suppose that PCTB's OCR-fed per-island data would be interesting to mine, but that's a bit too large a project for me to take on right now)
----------------------------------------
Pauling of the Lost Lot, Sagemerald
Officer emeritus and occasional cabin person
Puppetar (and drawing of secret Blackthorpe plans) by the talented Tilinka
[Apr 2, 2012 12:32:38 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
wrs1864b

Member's Avatar


Joined: Apr 10, 2006
Posts: 5450
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

 
Does anyone know what the ideal sampling interval is?

I don't know, but I know that the actual sales tax gets out of sync with my downloaded version in as little as 10 minutes, but I rarely am comparing stuff that closely and I could have just gotten unlucky a few times.

I'd be tempted to pick one ocean, download it every 5 minutes for a few hours or a day or two and see.

 
(I suppose that PCTB's OCR-fed per-island data would be interesting to mine, but that's a bit too large a project for me to take on right now)

Yarrg's downloader has an option to save the data locally, and I've been using that since late 2009. Yes, I find the market data to be far more useful than the tax data for trends. I would make my stuff public, but making it usable/understandable is a bit too large a projected for me to take on right now. :-<
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.
[Apr 2, 2012 1:29:28 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Pauling

Member's Avatar


Joined: Dec 20, 2005
Posts: 1997
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

I've changed the sampling interval from every 24 hrs to every three hours for now.

Incidentally: based on the above tip about BK tax rates, you actually CAN use BKs to get tax rates for every commodity at any time, if you have a spare island. On the emerald testing ocean, we discovered that BKs can be manually made governor of an island at any time, and then immediately voted right back out again by the monarch of the island holding flag. (But see this page before leaving PoE in the coffers, or otherwise prettifying the island:
http://yppedia.puzzlepirates.com/Brigand_King#What_happens_when_a_Brigand_King_takes_an_island )
----------------------------------------
Pauling of the Lost Lot, Sagemerald
Officer emeritus and occasional cabin person
Puppetar (and drawing of secret Blackthorpe plans) by the talented Tilinka
[Apr 2, 2012 3:32:35 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
LJAmethyst

Member's Avatar


Joined: Jul 19, 2007
Posts: 3653
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

Hmm, I wonder if you've obtained Three Rings' permission to scrape their websites like this - not sure if it is required, but it might be nice to ask if you're going to make regular dips.

Also, it might be interesting for you to check Jade and Opal as well; I don't know if there is any translation involved here.
----------------------------------------
Senior officer, Dynamite Residue.
Governor emeritus of Amity Island.
[Apr 2, 2012 4:34:25 PM] Show Printable Version of Post        Send Private Message    http://www.newadvent.org/bible/jon001.htm [Link]  Go to top 
Pauling

Member's Avatar


Joined: Dec 20, 2005
Posts: 1997
Status: Offline
Re: Monitoring commodity prices over time? Reply to this Post
Reply with Quote

 
Hmm, I wonder if you've obtained Three Rings' permission to scrape their websites like this - not sure if it is required, but it might be nice to ask if you're going to make regular dips.

Also, it might be interesting for you to check Jade and Opal as well; I don't know if there is any translation involved here.


That's an extremely good suggestion, and at the moment my only move has been to shy away from scraping too often (my goal is 1-2x a day if feasible). I think I dropped the ball a bit there. :) The information obtained is a trio of 9.8 kB publicly available pages, but I'll see about sending a petition this evening.

The file format appears to be the same for Opal and Jade, and the commodity names on the tax page weren't even translated... but let's make sure that scraping is ok before adding more oceans to the mix.

EDIT: Bia very helpfully pointed me towards the official policy on third-party tools, which may be found online. Looks like scraping is ok, within reason:
http://yppedia.puzzlepirates.com/Official:Third_Party_Software#Reading_public_ocean_websites
----------------------------------------
Pauling of the Lost Lot, Sagemerald
Officer emeritus and occasional cabin person
Puppetar (and drawing of secret Blackthorpe plans) by the talented Tilinka
----------------------------------------
[Edit 1 times, last edit by Pauling at Apr 2, 2012 6:53:57 PM]
[Apr 2, 2012 5:08:21 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
[Show Printable Version of Thread] [Post new Thread]

Puzzle Pirates™ © 2001-2014 Three Rings Design, Inc. All Rights Reserved.   Terms · Privacy · Affiliates