How I Built a Web Application to Draw Insights From Articles Across Web3 Publishing Platforms

Web3 publishing platforms are rising, and they bring new perks to the writing ecosystem. For instance, writers can sell their pieces as writing NFTs and receive cryptocurrencies in return. Web3 publishing platforms such as Mirror and Paragraph, solve some of the issues presented in the web2 counterparts. Platforms like Medium or Substack, can’t guarantee permanency for the articles, and they don’t solve the ownership issue. Web3 publishing platforms, rely on Arweave for instance, to save the data permanently in the permaweb. Each article is a JSON file registered forever with the author’s wallet name, which serves as a digital proof of ownership. Once submitted, articles can also be sold as writing NFTs, and people can buy them for several purposes.

Writing NFTs are new in the market, and they lack core features to make people acquire more. For instance, there are no good search engines, hence readers can’t easily find web3 articles by keywords, also there are no dashboards and data visualization tools, which could inform users about what is happening in the web3 publishing realm, without the need of losing countless hours reading or following the different projects on Twitter (aka X).

Some of the top articles in terms of collections and revenue can be really useful to follow or collect, since some of them introduce new projects, have giveaways and more. Post3 is a project that aims to explore Writing NFTs from web3 Publishing Platforms. Post3 publishes weekly insightful reports about what happened in the web3 realm. For that, it uses Web Scraping and GraphQL to extract data from Arweave. The data is cleaned and uploaded to the Ocean Market so other analysts can acquire the datasets and draw their own conclusions.

Until now Post3 has been delivering weekly reports and datasets of Mirror and Paragraph articles. While these reports provide information about new projects, airdrops, protocol updates, rewards and more, they can’t cover all the richness hidden behind the articles. This is why acquiring the datasets serves as the best approach to harness their full potential. You can find them in the link below.

Data Analysts, Data Scientists and data enthusiasts are familiar with data manipulation, and they can easily explore the datasets and grab the core information. However, most people just want to visualize the content they need without further work. That’s why Post3 is focused on building tools to upload the datasets and return meaningful outputs.

The first tool built is entitled Post3 Engine, and by uploading a single dataset, the user can generate more dynamic charts than the ones available in the weekly reports, and also obtain new plots and textual analysis. Besides, once with the dataset, users can make further exploration or save them for future Post3 applications.

In the following piece, we’ll delve into Post3 Engine, how to use it, and what are the plans for the future.

👉 Join Post3 Discord community here. Follow Post3 on Twitter (aka X) and Farcaster. Subscribe on Mirror

1 — The importance of web3 publishing data

As people start using more and more web3 publishing platforms, it becomes important to explore solutions to enrich this ecosystem. While some users are only focused on writing, others might be interested in hunting interesting content, finding the authors/publications that people liked the most, discovering strategies to optimize their writing, being informed about airdrops, rewards, new projects and much more.

Most of these perks can be achieved through Data Visualization and NLP (Natural Language Processing) tools. Post3 collects the writing NFTs from Arweave through GraphQL and applies Web Scraping techniques to obtain extra features that are not available at the time the article is submitted, such as the number of people that have collected, the revenue generated, the first publication date and more. The number of collections varies over time, that’s why Post3 waits at least one week to deliver the weekly dataset along with the report. The number of collections and revenue is not constantly updated but serves to make comparisons at a weekly rate and probably monthly or yearly in the future.

The Post3 Engine mentioned before, is the first tool developed by Post3, to make it easy for everyone to explore the weekly datasets. Enthusiasts can simply buy them with $mOCEAN on the Polygon chain, submit them on Post3 Engine and observe insightful dashboards that provide information about collections, revenue, networks and textual indicators.

Depending on the enthusiasm around the application, more dashboards will be introduced to make Post3 Engine even more powerful.

Meanwhile, you can use the Post3 Engine, the weekly reports and the datasets for the following:

  • Discover Airdrops and Rewards: Some of the most popular and lucrative articles cover airdrops. Submit a dataset to Post3 Engine or read the weekly reports to never miss an opportunity.

  • Discover New Projects: Almost every week, the top collections belong to a new web3 project. This is important because being the first to take part in a new web3 protocol/project is usually rewarding.

  • Find Writers: If you know some data analysis, you can import the dataset to a Python script, find content that suits your taste, and subscribe to the publications. Another way is to follow the weekly reports since there are always some Post3 picks.

  • Optimize Writing: With Post3 Engine you can apply textual analysis to the datasets and observe important parameters, such as the top keywords and the average body length of the articles. With this information, writers can have a general idea of the trends and apply strategies to reach more people.

  • Build: the datasets contain all the information of that week, you can build tools like Post3 Engine to make the most of this data, like creating ChatBots, data exploration, dashboards and more.

2 — The technical stack

The development of Post3 Engine was triggered by a Hackathon launched by the Ocean Protocol. This project owns several data-driven tools that can be used to make cryptocurrency predictions, compute algorithms, sell datasets and much more. With the help of Ocean.py, Post3 was able to create a script to automate the process of uploading datasets to the Ocean Market. Future development will allow users to connect to Metamask and buy the datasets directly from Post3 Engine, along with premium models drawing more advanced insights from the data.

The platform was built using mainly Python, HTML and CSS. For the backend, we used Flask, which is a Python framework that provides flexibility and seamless integration with other packages, such as Plotly Dash. The latter was used to make the widgets and dashboards. If you want to learn more about the development of Post3 Engine, I recommend you to read the following Medium article:

The project is also fully open source and available in the GitHub repository here.

It took around two weeks to put the project into production. More features are on the way, as well as a smoother experience to acquire the datasets and upload them. If there are models/features you would like to see in Post3 Engine, please report to the Post3 Discord community

👉 Join Post3 Discord community here. Follow Post3 on Twitter (aka X) and Farcaster. Subscribe on Mirror

3 — Explore the platform

Post3 Engine was meticulously crafted for intuitive operation. The user simply selects a dataset from the main page and uploads it along with a specific model. At the time I’m writing, there are two models, one that generates bar charts concerning the collections and revenue and a textual analysis model that prompts word clouds, comparison of language usage and more.

More features will be implemented in the future, but right now feel free to try the application by following the steps in the video below:

Conclusion

Post3 is always trying to figure out ways to get the most out of the datasets collected from web3 publishing platforms. We do believe they are a valuable source of information. With tools such as Post3 Engine, the web3 publishing landscape can evolve into something richer, where users and writers can have a wider vision of the articles being created and act accordingly.

Post3 aims to enrich the web3 publishing ecosystem in several ways, and it will flourish as long as there’s adoption of reports and datasets. As it is still in very early stages, we appreciate any recommendations and criticism to make our products better. The best way to reach out is by joining the Post3 Discord community.

To learn more about Post3's mission and how it intends to contribute to the web3 publishing ecosystem, please refer to the following entry:

Subscribe to Marco Rodrigues
Receive the latest updates directly to your inbox.
Mint this entry as an NFT to add it to your collection.
Verification
This entry has been permanently stored onchain and signed by its creator.