• Play
  • About
  • News
  • Forums
  • Yppedia
  • Help
Welcome Guest   | Login
  Index  | Recent Threads  | Register  | Search  | Help  | RSS feeds  | View Unanswered Threads  
  Search  


Quick Go »
Thread Status: Normal
Total posts in this thread: 17
[Add To My Favorites] [Watch this Thread] [Post new Thread]
Author
Previous Thread This topic has been viewed 6345 times and has 16 replies Next Thread
Tallinnamany



Joined: Jun 13, 2005
Posts: 17
Status: Offline

Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Couldn't find this having been suggested previously (though did only do a quick search) so here goes.

What?:
Would it be possible to add a way to export data from the game in a structured way (eg json/csv)?
Essentially, it'd just be a way to export the data that you can already see in your client in a clean and simple way.

As a start, it'd be amazing if this was an option on screens such as the market trading screen, shop/ship inventory (and records) and shop pricing screens, allowing you to quickly gather the relevant data.

Why?:
As it is right now, (as an example) running a shop efficiently usually involves a massive excel spreadsheet and a whole lot of manual work keeping the prices and such up to date. This can be quite off-putting/punishing for someone new trying to get started with running a shop.

Making it possible to easily grab the data would open the door for developers to jump in and create tool(s) to make it less of a mess to manage - As a developer myself I'm seeing that a 'shop managing tool' of some sort would be incredibly valuable, but the amount of manual work keeping it up to date is enough to put me off the idea to even get started. I wouldn't be surprised if there were other people/devs in a similar position.


--- Feel free to ignore beyond this point, got a bit rambly ---

Technical side (thinking as a java developer myself. May be completely wrong):
While I can't make assumptions about how the game client works internally (and 15 years of legacy code can give anyone nightmares), it makes sense to think that the relevant data is already available on the client side (as its being presented to the user). So I'd imagine this could be done all on the client side - Strip the objects of unwanted information, pass the collection through something like gson (converting the list to json) and write that to a file. But again, legacy code could make something seemingly simple nearly impossible.

Background/Why do I think this would be good/worth doing?:
After moving on from pirates many years ago, I got into Eve Online, a game that shares quite a lot of the 'meta' parts with PP (Huge social/community side of things, player driven economy, conflicts/wars over territory, even a nearly identical structure for player groups (crews/flags -> corporations/alliances)). And similarly to Eve, PP has a bit of a steep learning curve if you want to get into more advanced game play features.

To help with this in Eve, there is an ever growing community of 3rd party developers who continuously keep creating and improving tools that make the otherwise difficult to comprehend activities/things more easily digestible. But this is (more or less) only possible because CCP (the developers) have made a huge effort to make the in-game data/resources available for anyone to work with as well as working with the developer community to work towards making more useful things accessible.

This has, over the years, resulted in there being a tool for pretty much anything you could want/need, ranging from sites that provide useful information (eg for market data) to all sorts of planning tools (from your own character to managing complicated manufacturing setups) to tools that automate otherwise slow/manual tasks (for instance, automatically providing access to voice services to freshly joined members, removing permissions on people who have left).


Summary:
While hoping for PP to have a full blown api/rest services to query some data/static data dump is a wild dream and would take an unimaginable amount of work/effort, I think even adding ways to manually grab some data would be a big step towards getting more people involved with creating tools, which in turn would hopefully have a longer term benefit in getting more people over the initial 'curve' and interested in more complicated activities in the game.

TLDR: If you build it, they will come - Add a way to export data from game -> developer(s) will start creating tools to make use of the data -> hopefully get more people to take interest in the more complicated parts of the game (which would be good for the longer term health of the game).
----------------------------------------
Meridian/cerulean: Tallinnamany
Obsidian: Johka
Software developer - Someone lazy enough to invest time in making a tedious/complicated task less so.
[Oct 5, 2017 6:18:50 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
wrs1864b

Member's Avatar


Joined: Apr 10, 2006
Posts: 6314
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

 
Couldn't find this having been suggested previously (though did only do a quick search) so here goes.

I completely agree that this would be very helpful to others. Some of my recent requests for this kind of thing include this and that

 
Would it be possible to add a way to export data from the game in a structured way (eg json/csv)?

Personally, I prefer tsv because it is compact, and tabs don't show up in the YPP data, while commas and single/double quotes do.

 
As a start, it'd be amazing if this was an option on screens such as the market trading screen, shop/ship inventory (and records) and shop pricing screens, allowing you to quickly gather the relevant data.

Exporting public data such as the dockside and bid markets, the full tax data (e.g. pink paint used in bludgeons), the exact number of spawn points on a island, etc. should be fairly easy. Exporting things like the ship/shop records would require some kind of protection so that only certain people can see them. That could be a lot more work.


 
]Why?:

You are preaching to the choir here!

Not only do I see why, but I've actually done all the things you have suggested.

The game allows you to cut and paste the data. So, for example, I have a program that extracts all the records off the screen by copy and pasting the records. It automatically keeps hitting the "download more" button until I have a full list of new/modified records. I have a program that copies the hold information and commodity price configuration. The program that downloads the price modification screens for stores also gets the product recipes thanks to OOO's effort in providing them.

Even though I have programs to do all that, it still took me about 20 minutes to update my 7 stores and various ships. That is long enough that I would tend to only update stuff maybe twice a week.

The copy and paste method does not work for the dockside market though because every time something changes on the market (e.g. someone buys/sells or puts new stuff up), the screen resets and puts you on the top again. The copy and paste system is also not the most reliable thing and will frequently miss-copy lines. You have to put in a lot of sanity checks
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.
[Oct 5, 2017 7:26:35 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Missroxy

Member's Avatar


Joined: Jan 15, 2010
Posts: 3
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

I think this is a lovely idea!

I agree with you on helping people with the curve - also by allowing export of data it allows various applications (beyond java) such as pairing with discord bots and other devices. This could also easily lead to apps to support gameplay and beyond shops this could help with all sorts of trade markets etc.
[Oct 5, 2017 7:39:34 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Scarpath

Member's Avatar


Joined: Sep 7, 2014
Posts: 244
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Yes please! I am no code wizard myself- if I want anything, I have to rely on others. That means that the easier it is to make, the more likely it is to be made. And I have wanted shopkeeping tools for a looonnngg time. I mean, it's lucky that I'm not losing money! But I've never felt that the coding for such tools should go under GH, they really would need to be 3rd party.
----------------------------------------
Scarpath on Emerald, Cerulean, and Meridian!

Also Scarpath on Obsidian, Defiant to the core.

Your grammar should at least be as good as mine, take the time to make it decent!
[Oct 5, 2017 10:51:27 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Tallinnamany



Joined: Jun 13, 2005
Posts: 17
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

 
Personally, I prefer tsv because it is compact, and tabs don't show up in the YPP data, while commas and single/double quotes do.

Json would be nice as it'd allow for some resemblance of structure be retained (eg in case of recipes), but I think anything that could be exported as a whole would be a significant improvement over what we currently have (be it csv/tsv/json/xml/whatever).

 
Exporting public data such as the dockside and bid markets, the full tax data (e.g. pink paint used in bludgeons), the exact number of spawn points on a island, etc. should be fairly easy. Exporting things like the ship/shop records would require some kind of protection so that only certain people can see them. That could be a lot more work.

What I'm thinking as a minimum would be pretty much keep it as simple as possible - just a button to export what you can see, so for instance only a manager can see the shop 'books' -> only the managers could export stuff from the shop. You'd still have to go through each of the locations you want to export, but it'd be a handful of button hits instead of god knows how much copy-pasting.

Some public stuff could certainly be available on yoweb or some similar out-of-game accessible way, but this brings its own technical challenges so the complexity already goes up (and the chances of resources for this to be developed being available go down). It would be nice though if you could pull as much of the static data in some simple manner though (so things like taxes, delivery costs, commodities, recipes, island related stuff etc).

A dream would be to have some kind of a full blown api, to allow you to query pretty much anything you could see in game, but this comes with so much complexity that I wouldn't even dare to suggest it. A great presentation from this years Eve Fanfest on this - EVE Fanfest 2017 - ESI Does it. Goes over the history of the resources available for 3rd party developers and gives some idea on the complexity of whats on the server side of the whole thing.
----------------------------------------
Meridian/cerulean: Tallinnamany
Obsidian: Johka
Software developer - Someone lazy enough to invest time in making a tedious/complicated task less so.
[Oct 5, 2017 11:30:19 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
fishy55



Joined: May 29, 2011
Posts: 1
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

I agree completely - more so on the shop management front the actual interface has always been a bit of a hassle to manage and keep track of all the prices so being able to export would streamline the whole process
[Oct 5, 2017 12:02:08 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
BehindCurtai

Member's Avatar


Joined: May 25, 2004
Posts: 12589
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Yes, please.

This is kinda needed if I'm going to go back into stall running.

(Hmm, have any of my old stalls auto-piloted and stayed alive this long?)

(NB: If I was to really get back into stall running, it would take a major change to how labor worked since the last time I did. That's another post that I have yet to write up.)
----------------------------------------
"We're trying to find the error bars on that number"

Dylan wrote: 
Why buy sham poo when real poo is so readily available

[Oct 7, 2017 8:30:17 AM] Show Printable Version of Post        Send Private Message    http://StrictConstitution.BlogSpot.Com [Link]  Go to top 
Fendero



Joined: Jun 29, 2017
Posts: 11
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

I have written the following applications regarding commodity market data:

The URL below links to the program that can read market data from the commodity market table reliably using the Java Access Bridge. It can export to JSON aswell as CSV.

https://github.com/fabianPas/MasterMarketeer

The other application is a .NET Core ASP app that calculates a trade route between 2 commodity market exports.

https://github.com/fabianPas/TradeRouteMarketeer

If you have any questions, hit me up in-game. If you want to thank me with a donation, please go ahead. ;p
[Oct 8, 2017 11:30:11 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
wrs1864b

Member's Avatar


Joined: Apr 10, 2006
Posts: 6314
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

 
I have written the following applications regarding commodity market data:

The URL below links to the program that can read market data from the commodity market table reliably using the Java Access Bridge. It can export to JSON aswell as CSV.

Nice!

Is this information available to others like PCTB or YARRG, or is this only stored locally? And, does this work on Obsidian, or just the YPP classic oceans?
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.
[Oct 8, 2017 4:40:01 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Fendero



Joined: Jun 29, 2017
Posts: 11
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Currently only Obsidian. As you have to select the island manually. I (or someone that creates a PR) could add this easily to support more oceans though. I haven't found a way to abstract the current island name from the client yet while at the commodity market screen.

It doesn't support others, but I don't know if you can import/export data by file with PTCB? I think the most interesting application is the Puzzle Pirates Trade Profiteer. If this supports importing/exporting data I could create compatibility.
[Oct 8, 2017 11:58:23 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Sweatybobo



Joined: May 10, 2017
Posts: 1
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Heres a python script I used to use to scan the PV market: https://hastebin.com/fucemasihi.py (someone make a backup of that if you want, hastbins go away after a while)

Basically it just presses down and copies text until it gets to the bottom. Sometimes copying a line will fail so it retries until it gets new text in the clipboard. It scans the entire market in about 20 seconds I think.

Then it automatically uploads all the current highest prices to my google sheet:

https://docs.google.com/spreadsheets/d/147V16W1i3YSEehvOvBMUeUQeudPJRQsPVl7bUU7lO7Q/edit?usp=sharing

You can import the current tax rates into a google sheet with

=IMPORTHTML("http://obsidian.puzzlepirates.com/yoweb/econ/taxrates.wm","table",2


With that data you can automatically calculate the cost of production for every shop good with a single scan.

My sheets were set up like this.

I used this really ugly custom javascript function for the sheet that would take a recipe string and automatically lookup and sum the prices + tax of all the commodities.

You just need to set all your prices/taxes in a single column like this: https://imgur.com/Pg8lJhl

The post-stall cost columns are so you can have a setup where your apoth sheet the unit cost of hemp oil production from your distillery, and then your shipbuilding sheet references your apothecaries production cost for laquer. That way you always get the true cost of producing a good if you own the whole production chain. If you dont do multiple lines like that then you get circular references and it breaks.

So yeah thats how I wasted my summer.
[Oct 10, 2017 1:22:22 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Forculus
Developer
Member's Avatar


Joined: Mar 30, 2011
Posts: 396
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

I agree that having easy access to this kind of data on the client side would be nice and open the door to some cool stuff. The main holdup right now is just finding time in the midst of game changes and feature updates that we're working on. Even seemingly simple stuff can end up taking much longer than expected, so we don't want to delay higher priority stuff.

Another consideration is implementing it in a way that isn't confusing to the average player, since this would only be used by a very small portion of the player base. A lot of our panels are already crammed full of buttons and info, so it would be preferable to not add another button that's rarely ever relevant.

Tallinnamany wrote: 
Technical side (thinking as a java developer myself. May be completely wrong):
While I can't make assumptions about how the game client works internally (and 15 years of legacy code can give anyone nightmares), it makes sense to think that the relevant data is already available on the client side (as its being presented to the user). So I'd imagine this could be done all on the client side - Strip the objects of unwanted information, pass the collection through something like gson (converting the list to json) and write that to a file. But again, legacy code could make something seemingly simple nearly impossible.

This is a pretty good overview of the situation. I've been experimenting around with some gson options, and was getting circular reference exceptions when it tried to parse some of our objects. I've tried to work around this, and just put out a test update on Ice where you can double-click on the "Inventory" tab and it will append the data to a local log file. There will be some extra stuff in there that you don't need, but hopefully it's not a problem to parse through and just extract what you want. If this seems workable, I can see about adding the functionality to other panels within the game.

If you run into issues or there's missing data that you need access to, let me know and I can try to find time at some point to make the implementation more robust.
[Oct 11, 2017 1:58:49 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
HotNoob



Joined: Jul 15, 2007
Posts: 9
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

https://yppedia.puzzlepirates.com/Official:Third_Party_Software
"Third party software may read any text that is normally visible to players in the client while not playing a puzzle. This includes the data generated by selecting and copying trade and similar tables. For the sole purpose of making the appropriate tables and menus visible, software may make use of scroll bars and drop down menus while not playing a puzzle. Software may also automate selection and copying of text."

Just want to confirm i can use Fendero's thing. A method of collection that is different than other apps.

would not require devs to do create any api's or whatever, if we can use this java access bridge explorer. to read ui elements (outside of puzzles ofc).

win-win if we are allowed to use this method.

by the wiki definition, it's ok, but i just want to confirm. :P
[Oct 12, 2017 10:51:12 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Fendero



Joined: Jun 29, 2017
Posts: 11
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

The method I use, uses native Java functionality called the Java Access Bridge, which is a bridge to allow for screenreaders etc. to read out Java applications and create accessibility applications.

It has been used before in Third Party Software, so I definitely think it's allowed, but maybe Forculus could confirm. Anyway, I set up an issue to support multiple oceans instead of just Dark Seas.

Maybe we should create a seperate thread instead to create some sort of support for the applications. As it is an open source application I would like as many to contribute!

For example, the market bidding could still be extracted, and it would be nice to find a way to extract the island name automatically.
[Oct 13, 2017 12:57:24 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
wrs1864b

Member's Avatar


Joined: Apr 10, 2006
Posts: 6314
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

Forculus wrote: 
I've been experimenting around with some gson options, and was getting circular reference exceptions when it tried to parse some of our objects. I've tried to work around this, and just put out a test update on Ice where you can double-click on the "Inventory" tab and it will append the data to a local log file. There will be some extra stuff in there that you don't need, but hopefully it's not a problem to parse through and just extract what you want. If this seems workable, I can see about adding the functionality to other panels within the game.

If you run into issues or there's missing data that you need access to, let me know and I can try to find time at some point to make the implementation more robust.


Sagacious wrote: 
Here's an example of an export log file

It corresponds to this inventory

OK, now that I can see an example of the output, here are a few things:

1) The "main" data is in the form of a "type" number for the commodity, followed by the quantity of that commodity in the hold. The "type" numbers can be extracted via the copy and paste interface. Here is a nearly complete list of all the commodities and the type numbers:
http://www.schlitt.net/yohoho/commods_data.tsv
http://www.schlitt.net/yohoho/commods_data.tsv.format

2) There is a WHOLE lot of extra stuff that I don't immediately recognize what it is for.

3) I've been using the copy and paste method to extract the hold information for many years and I have around 200 scripts/programs that I have that I've written to do various things. It turns out that the hold information isn't that useful for the things I wanted to do. Going through all of my programs, the only one that I think others would find useful is my blockade management program where I look at the holds of all the blockade ships and print out which ships need how many more units of booze and balls and if any of then have stray commodities.

4) This is an area that I've been interested in from when I first started playing and I've watched it closely. I can't remember anyone talking about extracting hold info. The dockside market? oh yeah. The store configuration stuff? spreadsheets all over the place. the list of ships? yep. But, I'm the only one I know of who has ever regularly saved hold info.

Forculus, I hope you aren't disappointed if this test doesn't get used that much.

I still think that exporting the dockside market data and complete bid market information to yoweb would help a lot more people. And, by "complete bid market info", I mean list the volume of each commodity at each price level. This would give players a way to see under the highest bid and help close the exploit of letting one player monopolize the bid market by placing a bid for a single unit at a very high price.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.
----------------------------------------
[Edit 1 times, last edit by wrs1864b at Oct 19, 2017 6:53:43 PM]
[Oct 19, 2017 6:28:43 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
fire2009boy

Member's Avatar


Joined: Apr 20, 2006
Posts: 176
Status: Offline
Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

 
A lot of our panels are already crammed full of buttons and info, so it would be preferable to not add another button that's rarely ever relevant.


Offtopic for this thread but any chance we'll ever get a rework of the shoppe management interface. There are too many buttons and a couple could easily but consolidated to one button that opens a screen with multiple tabs. For example, move the dockside listings as a tab to the screen where you manage all your prices
----------------------------------------
Kenpachi
[Oct 19, 2017 7:32:38 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Jstark1991



Joined: Oct 13, 2006
Posts: 1
Status: Offline

Re: Exporting in-game data/features to support 3rd party development Reply to this Post
Reply with Quote

I would have to agree with everything here. I was excited to see this test coming out, but the export is just.... messy.

In fact, I was trying to bring myself to build something cool just to help add weight to the argument that we need this stuff, but it's less work to use the java access bridge library to extract the data than it is to use the data being provided.

I don't want to only complain... This is a brilliant idea that I feel could just use some tweaking and community input to help make it something brilliant for developing tools for puzzle pirates.

Here's a suggestion of what the json export could potentially look like:
Json Export Example

Exporting the vessel inventory information could be limited to the vessels the player owns if rank is less than fleet officer, or potentially add title/permission within a crew of 'Quartermaster' that would allow them to export their crews vessel inventory... Or just open it up to everyone, since they can go to each individual ship and see the inventory regardless of rank anyway...

Anyway, I'm beginning to ramble, but I wanted to input my thoughts and suggestions and thank the devs for hearing the community and putting in stuff like this to make it a better game!
[Oct 23, 2017 12:27:23 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
[Show Printable Version of Thread] [Post new Thread]

Puzzle Pirates™ © 2001-2016 Grey Havens, LLC All Rights Reserved.   Terms · Privacy · Affiliates