Puzzle Pirates Forums - View Thread - Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Welcome Guest | Login

| View Unanswered Threads

Forums » List all forums » Forum: Tips & Tricks, Questions & Answers » Thread: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Quick Go »

Thread Status: Normal
Total posts in this thread: 19

[Add To My Favorites] [Watch this Thread] [Post new Thread]

Author

This topic has been viewed 7570 times and has 18 replies

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Ahoy!

I want to publicly announce my intent to fork the project known as Pirate Commodity Trader with Bleach (PCTB). More specifically, the OCR client known as PPAOCR, as the source code for the web server was never released.

I am creating this thread incase any of the original creators of PCTB are still active and have any concerns. Also, to allow for discussion in the community.

The resulting project will be under the MIT licence , the same as the original.

Why?

A lot of trade tools currently require 'hooking' into the Puzzle Pirate client to access data from the commodity screen. My understanding is these tools use their own copy of Java with accessibility options turned on (screen readers, etc) to allow reading of the market data as plain text. However, with the release of the Puzzle Pirates - Dark Seas, I haven't found a way to get these tools to work with the Steam install of Dark Seas. As such, I will looking to revive PCTB.

Problems with the original
The current version of PPAOCR has a couple of issues which have caused it to stop working.

OCR doesn't work on some clients
Both the official YPP client from http://www.puzzlepirates.com/ and the new Dark Seas client have some changes to the header height in the commodity window, which means the OCR no longer works on those clients. It does still work with the Seam version of YPP.

Upload site no longer exists
The site at http://pctb.crabdance.com/ is no longer available. So in addition to fixing up the issues with the client: I will be looking into creating a server application that will also be released to allow anyone to host their own instance if they so desired.

When?

Unless:

The project would be a waste of time, because there is already a trade client that works with Dark Seas
The original project owners decide to continue the original project

I intend to create a fork of the project sometime in the next week. The project will be hosted on github.

In the meantime, please let me know any thoughts/concerns or any ideas for a project name!
----------------------------------------
Fatman - Obsidian

[Sep 20, 2017 5:18:05 AM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Why not use YARRG?

And, since Obsidian only has three colonized islands, two of which aren't used much, it really isn't critical that you need trading tools right now.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Sep 20, 2017 5:33:57 AM]

[Link]

Sagacious

Joined: Mar 6, 2006
Posts: 4798
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

As for the different table headers, that's likely going to relate to the version of java being used to launch the game.
----------------------------------------
Cerulean & Meridian - Icemeister
Emerald & Obsidian - incognito!
#TeamPurple

[Sep 20, 2017 5:53:13 AM]

[Link]

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

wrs1864b:

Why not use YARRG?

I have been unable to get YARRG/JARRG to work with Dark Seas:

JARRG runs the unmodified Puzzle Pirates game, but with a separate copy of
your system's JVM (Java Virtual Machine); that copy of the JVM is configured to
use the JARRG accessibility plugin. We do this (rather than configuring the 
computer's main JVM to use the accessibility plugin) to avoid interfering with other
uses of Java on your computer - in particular, so that it is always possible to launch
Puzzle Pirates without JARRG (for example, in case the plugin should cause any
kind of problem).

Because Dark Seas requires Steam to log on, and JARRG uses it's own copy of a JVM, I couldn't get them to work together.

If you have a way to achieve this, please let me know.

wrs1864b:

And, since Obsidian only has three colonized islands, two of which aren't used much, it really isn't critical that you need trading tools right now.

Of course, I agree it isn't critical, but it would be useful.

Sagacious:

As for the different table headers, that's likely going to relate to the version of java being used to launch the game.

Agreed. The current version of PPAOCR expects them to be a certain size though, which is why the screen scraper no longer works. Which is why it's my intent to make a copy of the project, to get it working again.
----------------------------------------
Fatman - Obsidian

[Sep 20, 2017 6:05:08 AM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

I have pretty much retired from the game, so I haven't investigated problems with YARRG much.

In my experience, the old OCR based system was much less reliable (and slower) than the JARRG uploader (which worked for both PCTB and YARRG). It required the YARRG maintainer to constantly update the OCR system to recognize new fonts and character sizes and even then, it would get things wrong at times. (For example, sometimes you would get listings for commodities from "Mercenary Supply Shop", "MercenarySupply Shop", AND "Mercenary SupplyShop" without any indication of which was the most current data.) The maintainer for YARRG actually tracked me down in-game to complain about me still using the OCR program long after the JARRG uploader was published because it created extra work for him to do.

If you are going to put some effort in to this, I highly recommend trying to get the JARRG uploader working in some form.

OK, not only does JARRG not work, but YARRG doesn't support Obsidian and you can't get YARRG to support Obsidian because no one has updated the Yppedia for all the island maps yet. I keep telling myself that I should do that, but, well, I'm mostly retired.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Sep 20, 2017 6:51:12 AM]

[Link]

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

In my experience, the old OCR based system was much less reliable (and slower)
...
If you are going to put some effort in to this, I highly recommend trying to get the JARRG uploader working in some form.

I actually spend wayyy too much time trying to get tools like JARRG and Trade Profiteer working with the new Steam client. I might spend some more time on this though because I do agree that the approach by YARRG is much more robust, but the main problems I encountered is replacing the java_vm in the Dark Seas directory means Steam sees the files you have locally doesn't match the files Steam has, so it causes it to patch the game.
----------------------------------------
Fatman - Obsidian

[Sep 20, 2017 8:16:53 AM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Fixing PPAOCR is just a fairly small part of cloning PCTB, and for the most part, PCTB doesn't have the functionality of YARRG. There are two good points about this. First, the YARRG creator(s) have released the full website source code. Second, I have developed my own system that is in many ways much better than YARRG. But, as I mentioned, YARRG currently can't support Obsidian because the Yppedia doesn't have all the island maps and my system depends on a working YARRG.

Being mostly retired from the game for about 2 years, I've been having a very hard time getting motivated to do the work needed to let others use the system I have developed. I'll try get something out soon so you don't have to re-invent the while for the 4th or 5th time.

The best solution to the market data upload problem is to have GH get off their butt and export the dockside market data and the bid market data via yoweb.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Sep 21, 2017 5:48:28 AM]

[Link]

Shinito

Joined: Jan 3, 2006
Posts: 588
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

The best solution to the market data upload problem is to have GH get off their butt and export the dockside market data and the bid market data via yoweb.

----------------------------------------
Thorkill on ~~Malachite 2.0~~ Obsidian

[Sep 21, 2017 8:34:37 AM]

[Link]

JerikTeloria

Joined: Jun 18, 2011
Posts: 1
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

I am brand spanking new to the community but I was actually looking to do something similar to this using Tesseract and C#. If you'd like to start from scratch it shouldn't take too long and I can share with you the code I have so far.

https://github.com/charlesw/tesseract

The best solution to the market data upload problem is to have GH get off their butt and export the dockside market data and the bid market data via yoweb.

That said, I do really think this is the better solution.
----------------------------------------
[Edit 2 times, last edit by JerikTeloria at Sep 21, 2017 3:21:29 PM]

[Sep 21, 2017 3:18:26 PM]

[Link]

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Thanks! Help would be much appreciated. I will look at getting the project on github later today.

I did look into Tesseract, but the current OCR client seems to be working fine. It may be an option moving forward?

Just for reference: While the current client is written in C# but I'm actually a Python dev (I'm currently working with IronPython, so I have had some experience with .NET), so I was intending on creating the web app using Python. I don't know if this is something you'd still be interested in?

That said, here are my current objectives:

Clear out some unnecessary parts of the current client.
The client should export the data to JSON.
Basic web app that accepts and stores the data, and allows it to be browseable.

----------------------------------------
Fatman - Obsidian

[Sep 22, 2017 1:53:12 AM]

[Link]

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Just a quick update.

Both projects are now on github:

PACS - Market Scraper
github.com/alxwrd/pacs

The market data portion of the client is working. I am debating if I should remove the bid data as I don't see how it's useful in a trading tool (you would need to sit on a bid ticket for a bit,
at which point the dock prices might have changed). I have done a release if anyone wants to try it out. The upload won't work as it currently has no where to go.

Instructions:

Download and run the latest release of PACS.
Open the trading window or bid window in Puzzle Pirates (either from an island or a ship).
Capture the data using the "Capture Market Data".

Data is stored as json in %APPDATA%\PPAOCR\Commodities.json and has the following format:


{
    "ocean": "OCEAN NAME",
    "island": "ISLAND NAME",
    "goods": {
        "Sugar Cane": {
            "buy_orders": [
                {
                    "shop": "SHOP NAME",
                    "price": 5,
                    "amount": 375
                }
            ],
            "sell_orders": [
                {
                    "shop": "SHOP NAME",
                    "price": 6,
                    "amount": 375
                }
            ]
        }
    }
}

Merchant Almanac - Web Server
github.com/alxwrd/merchant-almanac

Not totally sold on the name. Not much has happened here as I would have liked, but this is my next focus.

Going forward, I won't keep updating this thread with development as I don't want to keep bumping it. If you want to follow along, keep an eye on the github repos.
----------------------------------------
Fatman - Obsidian

[Sep 25, 2017 3:16:31 PM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

I am debating if I should remove the bid data as I don't see how it's useful in a trading tool (you would need to sit on a bid ticket for a bit,

Personally, I found the bid market information to be more important than the dockside market data. Since the JARRG uploaded didn't keep track of it, I used other techniques. While bid tickets aren't as immediate as gems or dockside trades, filling tickets and then shipping around the filled commodities are where the real PoE is at. That said, if you are an experienced trader, you can almost keep track of the bid market in your head. What hiding the bid market does it it doesn't open the eyes of inexperienced traders.

The other comment I have is that I would f recommend is keeping a timestamp for when the data was uploaded.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Sep 26, 2017 5:08:28 AM]

[Link]

alxwrd

Joined: Sep 19, 2017
Posts: 7
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Just want to drop in to post a quick note about potentially using JARRG going forward. I was looking through the Terms of Service to see if I can use in game art/assets and on the page about Third Party Software there's the following section:

Modifying game files

Modification of any files in the Puzzle Pirates directory is not permitted with one exception: Players with a vision deficiency may replace puzzle graphics for the sole purpose of bringing their puzzle tile recognition on par with normal vision. A more informal explanation can be found here .

Previously JARRG had it's own Java vm. But because Dark Seas runs through Steam to get Steam integration you would need to modify the Java vm in the directory, which wouldn't be allowed.

I don't know if we could get an OM comment on this?

P.s. Does anyone know what the stance is for using assest in 3rd party tools?
----------------------------------------
Fatman - Obsidian

[Sep 28, 2017 12:50:16 PM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

I don't know if we could get an OM comment on this?

P.s. Does anyone know what the stance is for using assest in 3rd party tools?

On this kind of thing, when in doubt, do a petition. Make sure you are very clear about the situation and that probably a dev not just an OM needs to consider things.

That said, I recommend looking at the Official:Third Party Software. When this policy was last updated,YARRG (and PCTB) was explicitly discussed with a reply from the OM Endymion.

So, I'm pretty sure that GH knows what the JARRG uploader does and is ok with it. The important thing is that the JARRG uploader neither modifies any file, nor does it modify the game executable either on disk nor in memory. Early on, JARRG used a different run time java executable on disk, but OOO convinced the JARRG developers that they need to use the standard java program that YPP uses and use the standard "screen reader" API to do all the work.

So, does this reasoning still apply to Steam? I don't know, but I suspect GH will say it is OK.

For what it is worth, I host an updated version of the JARRG uploader that kind of works with YPP classic. The source code on the YARRG web site broke when OOO changed the game. Now my patched version started to fail when a new version of java was released and I've never gotten around to fixing it. I don't think I've ever published the patches I made, I'll try to fix that.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Sep 29, 2017 6:33:01 AM]

[Link]

polaris395

Joined: Oct 23, 2005
Posts: 1330
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

SO I've been scraping with it and it has been grabbing the data well. Unfortunately, I was able to CTRL+C into an excel document (great feature), but now I can't do it again. I select all the data, copy it, and then all the rows of data clear.

Any ideas on whether the export to excel can be made easier? Using the clipboard would be ideal.

Thanks!

Edit: Also, I am unable to get PV to load. It says island not found. Lionhaven loads however.

Win 7 Ultimate, 64 bit.
----------------------------------------
-Polariss

Apollo wrote:

I saw Apollo do it...maybe a week ago? He was by the SF table.

You mean "I saw Apollo in the Lima inn - perhaps he did it", and the answer is "No, I didn't".

----------------------------------------
[Edit 1 times, last edit by polaris395 at Oct 17, 2017 7:33:05 PM]

[Oct 17, 2017 2:48:45 PM]

[Link]

Fendero

Joined: Jun 29, 2017
Posts: 11
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

You might want to take a look at my scraping tool, which is open source and doesn't require OCR at all.

http://forums.puzzlepirates.com/community/mvnforum/viewthread?thread=207575

[Oct 17, 2017 11:41:43 PM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

You might want to take a look at my scraping tool, which is open source and doesn't require OCR at all.

http://forums.puzzlepirates.com/community/mvnforum/viewthread?thread=207575

I will say again, that instead of re-inventing the wheel, people should look at YARRG. It is open source, doesn't require OCR at all, figures out the correct ocean and island by itself and does a whole lot more than just figuring out trades between two islands.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Oct 18, 2017 3:48:09 AM]

[Link]

polaris395

Joined: Oct 23, 2005
Posts: 1330
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

You might want to take a look at my scraping tool, which is open source and doesn't require OCR at all.

http://forums.puzzlepirates.com/community/mvnforum/viewthread?thread=207575

Ive seen it. Havent bothered to get visual studio, and lack of an executable makes laziness an issue. Since it outputs a CSV Ill look into it.

You might want to take a look at my scraping tool, which is open source and doesn't require OCR at all.

http://forums.puzzlepirates.com/community/mvnforum/viewthread?thread=207575

Again, seen it. Does it output a CSV? I dont really care about sharing data with others. I just need to bring all market data into excel so I can do my own thing. Looking at the website clearly Im using it wrong because I cant get any trade routes on Obsidian to actually be made. But thats an issue for that thread anyway.
----------------------------------------
-Polariss

Apollo wrote:

I saw Apollo do it...maybe a week ago? He was by the SF table.

You mean "I saw Apollo in the Lima inn - perhaps he did it", and the answer is "No, I didn't".

[Oct 18, 2017 7:04:13 AM]

[Link]

wrs1864b

Joined: Apr 10, 2006
Posts: 6337
Status: Offline

Re: Intent to fork: PPAOCR (Pirate Commodity Trader with Bleach)

Reply to this Post
Reply with Quote

Again, seen it. Does it output a CSV? I don?t really care about sharing data with others. I just need to bring all market data into excel so I can do my own thing. Looking at the website clearly I?m using it wrong because I can?t get any trade routes on Obsidian to actually be made. But that?s an issue for that thread anyway.

The JARRG uploader has an option to write a "test" file of the island you uploaded, which I seem to remember being in tsv format, not csv. Excel will import either. The author is a strong supporter of open source programs and data, so it was designed to share the data too.

As far as why you can't get any trades on Obsidian via YARRG, well, no one has uploaded any island data yet. If you want to see what YARRG can do, look at other oceans.
----------------------------------------
Algol can not assert the truth of all statements in this post and still be consistent.

[Oct 18, 2017 7:24:50 AM]

[Link]

[Show Printable Version of Thread] [Post new Thread]