NPM (node.js library) version of my tree diagram generator, and a new interactive demo site

Posted by jimblackler on Jan 8, 2023

In 2014 I published a tree diagram generation tool (link). I’ve had a modest amount of positive feedback over the years so I decided to update it to use modern JavaScript (ES6) plus TypeScript  and package it as an NPM project. It’s amazing how much the web world has moved on in the last eight years, and giving something a polish can be surprisingly satisfying.

The differences were so significant that I felt a new GitHub repo was appropriate. It’s here.

I’ve also developed a new online version of the tree generator that can act as a diagram generator, demo and playground for the tool. It has a new IDE-like design, thanks to the excellent GoldenLayout library. It’s here

Once again the tool is free software offered under an Apache 2.0 licence. As always, any questions can be reported as an issue on GitHub, or by emailing me at jimblackler@gmail.com.


Posted in Babble || No Comments »

Nonogram Pro; a puzzle game written in ES6+Typescript for node.js and the web

Posted by jimblackler on Nov 19, 2022

I’ve made a new project which is online in an App Engine instance at https://nonogrampro.appspot.com/

About

I’m a fan of Nonogram puzzles, also known as Picross and many other names. Nonogram puzzles were invented in 1987 by Non Ishida and independently by Tetsuya Nishio. Players have to complete a grid to discover which cells are shaded based on number clues in the rows and columns. The Wikipedia article on Nonograms is a good source of history of the puzzle type as well as information on how to play.

There are many Nonogram games that can be played online. However some may know that I made a website based Solitaire game which has become modestly popular over the years. https://jimblackler.net/blog/?p=303 People have said they appreciate the clean design with no advertisements or flashy effects that distract from the game. I thought maybe I could do the same for Nonogram puzzles. 

How Nonograms are completed

Completing the project

First something of a confession; I actually started this project in 2017. I made a lot of progress, but I must have got distracted by something, so I left it unfinished. Sadly as many tinkerers like me have found it’s a lot easier to start something than it is to finish. This project was still in the back of my mind though, so I made the decision to pick it up again recently, and have successfully completed it. This just shows it’s never too late to resurrect an abandoned project.

I did update it to the latest tech though. It’s now written in Typescript, and I converted the old Python back end to one written in Node.js. This is the tech that I’m really enjoying working with these days. One advantage of using Node.js is that code can be easily shared between client and server and that can be useful for puzzle processing. This is handy for the ‘hint’ function I’ve supplied with the game; it uses the same solver that is used server-side to make sure that puzzles can be completed.

How it works

The client works by rendering the puzzle as an SVG created in script.I could have used Canvas, but using SVG handles the graphical updates efficiently and allows me to use CSS to style the puzzles.

Having written the game I needed some actual puzzle designs to use. I could probably have done some drawings but I’m not really that artistic and I wanted about 50 or so. I hit on the idea of creating puzzles using libraries of icon images that can be freely remixed; i.e. ones that have a Creative Commons or public domain licence. Even better if they were stored in a vector format like SVG; they could be then scaled to the Nonogram grid.

This approach worked surprisingly well. My importer can iterate through whole libraries and render the images in a very large grid. It then ‘trims’ off the vertical and horizontal edges that contain no content, then shrinks the leftover image to a pattern of shaded cells of the required grid size.

However you might be surprised to learn that more often than not importing an image like that would create a puzzle that either had more than one solution or that would require advanced solution techniques because they couldn’t be solved one row or column at a time. That’s no fun for the player, so the importer actually attempts to solve the puzzle first from the generated clues. If it’s not possible to complete it using the standard set of techniques the puzzle isn’t used. If it is possible to complete it, the number of rounds required to complete with the standard method becomes the ‘difficulty’ shown with the puzzle.

How graphics can be converted into puzzles.

I then look at the generated puzzles and select the best ones to publish on the front page.

I tried to include a variety of grid sized puzzles, 5×5, 10×10, etc. up to 30 x 30. And the import worked so well that I smashed my target of 50 puzzles. There are over 600 in the collection.

Game and editor

The game uses the browser’s local storage to keep track of which puzzles you’ve completed. 

I’ve even included an editor so you can design and publish your own Nonogram puzzles!

You can try them in your own browser, or publish them on the site. This creates a link for the puzzles that could be sent to other people so they can try them. To do that requires signing in with a Google account and selecting an author name for your puzzles.

If you have generated any puzzle you’re particularly pleased with, let me know by email and I’ll consider adding them to the selection of puzzles that everyone sees on the front page of the site.

As usual the code for my projects is on GitHub so that anyone can see how it works, and made available with an open source licence.


Posted in Babble || No Comments »

JSON to Google Sheets importer written in JavaScript and Java

Posted by jimblackler on Jun 13, 2021

I’ve written about how much I like JSON before. It’s a really convenient way to store and transmit data across many different applications.

I also like Google Sheets, but while you can import CSV data, it’s not easy to import JSON data. This is a shame because I use tools that output data in JSON that would be really convenient to view in Sheets.

I didn’t like any of the workarounds I could find on the web, so I wrote my own. To make it as easy as possible for users, it uses the JavaScript Google Sheets API which means that the whole app can run in a website easily, and my App Engine site doesn’t have to deal with any user data (Authentication is done on the server because the client side authentication isn’t working well with browsers that prohibit third party cookies.) The application is already online, hosted on App Engine at https://jsonworkspace.appspot.com/

The main requirement is that the data is formatted as a JSON array. Each element will be converted to a Google Sheets row, with the columns set from the dictionary keys found in object. For example, pasting the below code into the form..

[
  {
    "Title": "The Godfather",
    "Year": 1972,
    "Stars": ["Marlon Brando", "Al Pacino", "James Caan"]
  },
  {
    "Title": "Gone With the Wind",
    "Year": 1939,
    "Stars": ["Clark Gable", "Vivien Leigh"]
  },
    {
    "Title": "Jaws",
    "Year": 1975,
    "Stars": ["Roy Scheider", "Robert Shaw", "Richard Dreyfuss"]
  }
]

.. would result in the following Google Sheet being created for the user when the Import button was pressed.

You can see in the above example how an embedded array (”stars’) was converted into named sub-columns in a process I call flattening. A similar process is used on dictionaries. This means that a variety of JSON documents can be imported; the importer is always able to convert the data to a 2D collection of cells. The only limit is the Google Sheets limits on the number of rows and columns.

I hope people find it useful. Feel free to contact me at jimblackler@gmail.com with any queries or comments.

The source is offered under an Apache License 2.0 and is available at https://github.com/jimblackler/jsonworkspace


Posted in Babble || No Comments »

jsonschematypes: a Java code generator from JSON Schemas

Posted by jimblackler on Nov 7, 2020

I like to develop apps that work on different platforms, and across different platforms. These apps often need to transmit data across the network (e.g. in the case of a web application with both server and JavaScript code), store data in files, or read configuration data. I’m a fan of using JSON for these types of applications. It’s a popular meta-format based on JavaScript types and syntax, but its use extends well beyond JavaScript applications. There are many interchange formats, but JSON is very well supported; there are JSON libraries for every language you can think of. It’s stored as text, which means it is easily sent in web requests, and it’s easy for humans to read and edit (much friendlier than XML, or a binary format, in my opinion).

A JSON file to store or transmit a player details in a game might look like this:

{
    "name": "Jim",
    "score": 500,
    "alive": true
}

On thing that makes JSON easy to use is that you don’t need a formal definition of valid data. One system can write a JSON file and another can read it, without any other files needing to be shared. The convention as to how the JSON should be interpreted is a matter of convention; determined by how the programs are written.

That’s great for making a quick start when you’re creating software. But it can cause a problem as the program grows in complexity, or more engineers get involved. Without a formal written structure (known as a schema) it’s easy to become confused about exactly how to read or write it.

JSON Schema

Fortunately there is a standard schema format for JSON, called appropriately JSON Schema.

A JSON Schema is itself a JSON document that specifies a test as to whether another JSON file passes certain tests that assert its validity for a certain application. An application developer supplies a schema for their program’s data, and if an individual file passes the schema, it’s valid for use.

This is one schema to validate the above JSON:

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "properties": {
    "name": {
      "type": "string"
    },
    "score": {
      "type": "integer",
      "minimum": 0
    },
    "alive": {
      "type": "boolean"
    }
  },
  "additionalProperties": false,
  "required": ["name", "score"]
}

I’ve been working on an application that uses JSON extensively. Much of the system is written in Java, so I use the popular org.json library to read the data. The text of a JSON file is parsed into an org.json.JSONObject or JSONArray, then it’s read using string key accessors. For example:

  public static void read(JSONObject jsonObject) {
    String name = jsonObject.getString("name");
    int score = jsonObject.getInt("score");
    boolean alive = jsonObject.getBoolean("alive");
    // ....
  }

Having to use string key accessors means that there’s room for programming error; even when a Schema is available. I found a few libraries that could copy JSON data into structured Java classes. However they all required the whole JSON file to be copied into Java classes at once. This means that if schemas aren’t available or practical for part of the data, they can’t be used. If the schema uses features that the library doesn’t support, it can’t be used either.

I wanted something that could allow me to gradually adapt a program written for org.json types into structured types, with the option to ‘fall back’ to regular unstructured data access for any reason.

Structured JSON access in Java

Something like this:

  public static void read(JSONObject jsonObject) {
    Player player = new Player(jsonObject);
    String name = player.getName();
    int score = player.getScore();
    boolean alive = player.isAlive();
    // ...
  }

This approach greatly reduces the changes of misinterpreting the structure. IDEs such as those published by JetBrains will be able to make suggestions as you type that match the intended format of the data you’re reading. In short; errors that would normally become evident at run time are made evident as you first write the code.

jsonschematypes

I wrote a library to generate Java wrappers around org.json objects that make this kind of structured access possible. It means that many format problems will be detected at build time, rather than run time.

It converts standard JSON Schemas into strutured access classes that wrap the JSONObject. They look like this:

public class Player {
	private final JSONObject jsonObject;

	public Player(JSONObject jsonObject) {
		this.jsonObject = jsonObject;
	}

	public JSONObject getJSONObject() {
		return jsonObject;
	}

	public String getName() {
		return jsonObject.getString("name");
	}

	public int getScore() {
		return jsonObject.getInt("score");
	}

	public boolean isAlive() {
		return jsonObject.getBoolean("alive");
	}

	public boolean hasAlive() {
		return jsonObject.has("alive");
	}
}

Design

Classes are created according to the design of the JSON Schema from which they are derived. The library takes some license in creating Java classes that match the inferred intent of the schema, to make them relatively lightweight and not overwhelming humans to read. This means tactcialy omitting some varations of accessors that could potentially be supplied. For example, if a schema specifies a default value for a property, no has method will be generated to test for the presence of that property on an object, since a value can always be returned. The effect of this is that changes to the schema could result in methods being removecd from generated Java classes.

Where the generator cannot identify any useful accessors to build, no class is generated and the containing class will supply JSONObjects directly.

The generated classes are designed to be as close as possible to idiomatic Java as possible. For example, using is getters such as isEnabled when exposing booleans.

They are not designed to hide or completely abstract the fact that the objects they interface are backed by JSONObjects and JSONArrays. It is a goal of the library that programmers can switched to unstructured access (via the usual JSON accessors) of data where required.

To build the classes I used a code generation library com.helger.jcodemodel which is an extended fork of Sun’s Java code generator.

As well as the Java library there’s a Gradle plugin that means it can be use in IntellIJ IDEA and Android Studio to automatically generate the classes when your schema changes.

Library, plugin and demonstration

Instructions on how to use it are on the github repository, where I’ve made the source code available under the Apache 2.0 license. Note there are separate instructions for the library and for the Gradle plugin.

There’s also an online demonstration where you can copy your own schemas into an editor and preview the Java code that would be created.

If you have questions, as always, feel free to contact me on the comments below, in GitHub issues on the project, or by email at jimblackler@gmail.com.


Posted in Babble || No Comments »

A Reddit Bot written in Python, using the Reddit and YouTube APIs

Posted by jimblackler on Nov 30, 2014

I regularly read Reddit on a phone, and I’ve come to admire a particular bot autowikibot. When someone posts a link to a Wikipedia article the bot replies with an excerpt from the article directly into the conversation. Without the bot replying, in order to understand why the link was posted I’d have to follow the link – taking me out of the app, incurring a delay and data use.

I noticed that there was a similar problem to be solved with YouTube links. Reddit users regularly post these but unlike Wikipedia links there’s nothing in the URL to indicate what the article might be about, just a string of digits such as “dQw4w9WgXcQ”. When I’m reading a conversation on Reddit and someone posts a YouTube link without explanation it’s frustrating; I have to leave the Reddit app to know why that post was made.

I built a bot in Python using the Reddit API (via the superb PRAW), the Google API for YouTube for video statistics, all hosted on the cloud application platform Heroku.

bot2

It’s up and running now, and can be seen in action at http://www.reddit.com/user/youtubefactsbot

What is Reddit and what are bots?

Redit is a massive and hugely popular discussion website. It has hundreds of millions of users, and thousands of subreddits (discussion pages). As well as internet users talking amongst themselves, the Reddit API allows the creation of ‘bots’. These can look like normal Reddit accounts, but their activity is controlled by an automated processes. The bots join in conversations; they typically react to a phrase and reply in order to provide information or amusement.

How does it work?

mobile

A bot is really nothing more than a manually-registered Reddit account being controlled through the API by a long-running program on a computer somewhere. Comments are fetched and and analysed by the program; if it chooses to reply, it does so through a POST to the API.

PRAW makes this process very easy with a helper method called comment_stream(). This allows you to get a look at submissions and comments as they are posted. Provided not much extra processing is needed, it’s feasible to keep up with the comment stream and react to every post.

My bot simply runs a regular expression over the comment to extract YouTube links, gets the video ID and fetches the data from the YouTube API. Most of the logic in the app is around formatting the comments and obeying Bot Etiquette.

Bot etiquette

From the outset with this project I wanted to ensure that the bot would be found useful and not annoying by Reddit users. It’s important to remember that a bot is a machine process injecting itself into a human conversation. This is one reason why bots have a mixed reputation on Reddit, even though subreddit moderators can choose to ban individual bots (very likely if they cause annoyance). I took care to err on the side of caution with the bot’s interaction.

At the time of writing, the bot:

In addition, the creator of autowikibot made not only the source for his bot online, but also (via Reddit’s wiki feature) the user and subreddit blacklist. These are users who have requested the bot not reply to them, and subreddits that have banned the bot. By applying the same blacklists to the youtubefacts bot from day one I was able to reduce the risk that the bot would comment where it wasn’t wanted.

Mainly I took care to make the information posted by the bot about the videos as information-dense as possible in order to justify its position in the threads. I have a ton of information available from the API, but a lot of it (such as bit rate, comment count and more) simply would be interesting enough to justify it’s place in the thread. I decided not even to include the channel (YouTube poster) name. I include the video name, running time, view count and posting date. I also include the first line of the description as it often adds useful information about the context. I don’t do this if it contains a link so as not to potentially introduce spammy links into Reddit threads, and also because those kinds of comments tend to be promotional rather than informative.

Source

The source is also online at https://github.com/jimblackler/youtubefactsbot and licensed under GPL.

If you want your own implementation you’ll have to register applications on both Reddit and Google API (YouTube), sign into both accounts locally, and upload the secrets and tokens folder to your application on Heroku.

Hope you like the bot.


Posted in Babble || 2 Comments »