Planet Squeak

blogs about Squeak, Pharo, Croquet and family
planet squeak - planet squeak es - planet squeak jp - planet croquet - planet squeak code - planet smalltalk

November 14, 2018

Torsten Bergmann

What FP can learn from Smalltalk

by Torsten (noreply@blogger.com) at November 14, 2018 04:32 PM

November 13, 2018

PharoWeekly

Better management of encoding of environment variables

Hi all,

Thanks Ben for reading.

For those wanting a follow up, I’ve proposed this pull request:
https://github.com/pharo-project/pharo/pull/1980.
I’m still working on avoiding dependencies against UFFI, fixing one other
test.
This is however almost finished, and given that I had to adapt the
original *abstract
proposal* to fit the real system, here is an updated version:

API Proposal for OSEnvironment and friends
=========================

OSEnvironment is the common denominator for all platforms. They should
implement at least the following messages with the following semantics:

– *at: aVariableName [ifAbsent:/ifAbsentPut:/ifPresent:ifAbsent:]*

Gets the String value of an environment variable called `aVariableName`.
It is the system reponsibility to manage the encoding of *both arguments
and return values*.

– *at: aVariableName put: aValue*

Sets the environment variable called `aVariableName` to value `aValue`.
It is the system reponsibility to manage the encoding of *both arguments
and return values*.

– *removeKey: aVariableName*

Removes the environment variable called `aVariableName`.
It is the system reponsibility to manage the encoding of *both arguments
and return values*.

API Extensions for *Nix Systems (OSX & Linux)
=========================

Since *Nixes environment variables are binary data that could be encoded in
any encoding, the following methods provide more flexibility to access such
data in the encoding of the choice of the user, or even in binary form.

– *at: aVariableName encoding: anEncoding
[ifAbsent:/ifAbsentPut:/ifPresent:ifAbsent:/put:] / removeKey:**
aVariableName
encoding: anEncoding*

Variants of the common API from OSEnvironment.
The encoding used as argument will be used to encode/decode *both arguments
and return values*.

– *rawAt: anEncodedVariableName encoding: anEncoding
[ifAbsent:/ifAbsentPut:/ifPresent:ifAbsent:/put:] / removeRawKey:*
*anEncodedVariableName*

Variants of the common API from OSEnvironment.
These methods assume arguments and return values are encoded/decoded by the
user, so no marshalling or decoded is done by it.

Rationale
=========================

– Encoding/Decoding should be applied not only to values but to
variables names too. In most cases Ascii overlaps with utf* and Latin*
encodings, but this cannot be simply assumed.
– Windows requires calling the right *Wide version of the functions from
C, plus the correct encoding routine. This could be implemented as an FFI
call or by modifying the VM to do it properly instead of calling the Ascii
version.
– Unix FileSystems and environment variables could mix strings in
different encodings, thus the flexibility added by the low level *Nix
extensions.

Other Implementation Details
=========================

– VM primitives returning paths Strings should be carefuly managed to
decode them, since they are actually C strings (so byte arrays) disguised
as ByteStrings.
– Similar changes had to be applied to correctly obtain the current
working directory in case it is a wide string.

 

Guille

by Stéphane Ducasse at November 13, 2018 06:55 PM

November 11, 2018

PharoWeekly

New random generator: WELL512 PRNG

Hi,

By accident I came across a pseudo random number generator that I never heard off before. It is supposed to be pretty good and had a very simple implementation. So I ported it to Pharo.

<class comment>

I am RandomWELL512, a random number generator.

I use the PRNG (Pseudo Randon Number Generator) WELL (Well equidistributed long-period linear) with a 512 bit state.

  https://en.wikipedia.org/wiki/Well_equidistributed_long-period_linear
  http://www.iro.umontreal.ca/~lecuyer/myftp/papers/wellrng.pdf

Implementation algorithm (See #nextUInt32)

  http://www.lomont.org/Math/Papers/2008/Lomont_PRNG_2008.pdf (Chris Lomont, www.lomont.org)

Usage

  RandomWELL512 new in: [ :r | (1 to: 10) collect: [ :i | r nextUInt32 ] ]. 

  RandomWELL512 new useUnixRandomGeneratorSeed; in: [ :r | (1 to: 10) collect: [ :i | r next ] ]. 

  RandomWELL512 new in: [ :r | (1 to: 10) collect: [ :i | r nextInt: 1000 ] ].

  RandomWELL512 new in: [ :random | | count all |
    random useUnixRandomGeneratorSeed.
    count := 1024 * 1024.
    all := Array new: count streamContents: [ :out |
      count timesRepeat: [ out nextPut: random next ] ].
    { all min. all max. all average. all stdev } ].

  [ RandomWELL512 new in: [ :random | 1 to: 1e6 do: [ :i | random next ] ] ] timeToRun.

Note that you should create one instance, seed it properly, and keep using it.

</class comment>

It is acceptably fast, generating 1M [0,1) Floats in about 0.1s. I compared the output with a fixed initial state to the C code that I started from and I got the same numbers. Maybe some people find this interesting.

I attached a file out.

Sven

by Stéphane Ducasse at November 11, 2018 08:53 AM

Zn nicer API

Hi,

I added a new convenience method to Zinc HTTP Components: ZnClient>>forJsonREST. This configures a ZnClient (HTTP client) to talk to standard JSON REST web services. Here are a couple of examples:

ZnClient new
  forJsonREST;
  get: 'https://jsonplaceholder.typicode.com/users'.

What #forJsonREST does is 3 things: set the 'Accept' header to 'application/json', install a #contentReader that parses incoming JSON as well as a #contentWriter that generates JSON.

ZnClient new
  systemPolicy;
  forJsonREST;
  url: 'https://jsonplaceholder.typicode.com/posts';
  contents: { #foo->1. #bar->2 } asDictionary;
  post.

As you can see, the full ZnClient API can be combined when needed.

ZnClient new
  forJsonREST;
  post: 'https://jsonplaceholder.typicode.com/posts' 
  contents: (NeoJSONObject new foo: 1; bar: 2; yourself).

#post:contents: combines separate #url: #contents: and #post message.

#forJsonREST uses NeoJSON[Object|Writer] if found, else STONJSON. If both are missing, this results in an error.
	
ZnClient new
  systemPolicy;
  forJsonREST;
  url: 'http://easy.t3-platform.net/rest/geo-ip';
  queryAt: #address put: '81.83.7.35';
  get.

Finally, here is a more sophisticated example, doing a DNS request over HTTPS:

ZnClient new
  systemPolicy;
  beOneShot;
  forJsonREST;
  accept: 'application/dns-json';
  url: 'https://cloudflare-dns.com/dns-query';
  queryAt: #name put: 'stfx.eu';
  queryAt: #type put: #AAAA;
  get.

Note that in most cases, you will configure one client to a specific endpoint and keep on reusing it. At one point in time it might be good to #close the client (although that happens on finalise as well). For single requests, you can use #beOneShot.

All this can be found in #bleedingEdge (HEAD). There are unit tests as well.

Sven

by Stéphane Ducasse at November 11, 2018 08:50 AM

November 09, 2018

Clement Bera

64 bits Immediate Floats

Hi all,

In this post I will try to discuss some inner details of OpenSmalltalk-VM immediate floats. Immediate floats are present only in 64 bits hence I won’t talk about 32 bits VM in the whole blog post. In addition, OpenSmalltalk-VM supports only double precision IEEE floating pointer, hence I won’t discuss single precision IEEE floating pointer.

Immediate objects

OpenSmalltalk-VM uses an immediate object scheme to represent object oriented pointers (oop) in memory. Basically, due to 64 bits alignment, the last 3 bits of all pointers to objects are 000. This is abused to encode in the oop itself specific objects, in our context, SmallIntegers, Characters and ImmediateFloats. This optimization allows to save memory and to improve performance by avoiding boxing allocation for common arithmetic operations. The last 3 bits of an oop are called a tag. The Immediate Float tag is 100 (4 in decimal). Objects encoded directly in the oop are in our terminology called immediate objects.

Immediate

Immediate floats

OpenSmalltalk-VM and its clients use the double precision IEEE format to represent floating pointers, supported by most modern hardware.

doubleIEEE.png

The key idea to the immediate float design is to use an immediate representation of double precision floats to avoid boxing and save memory, while still being 100% compatible with the IEEE double precision format (Customers requirement).

Therefore, in 64 bits, OpenSmalltalk-VM use two implementations for floats. The most common floats are represented with immediate floats, where 3 bits of the exponents are abused to encode the tags. The rest of the floats are represented as boxed floats.

ImmFloat

By design, immediate floats occupy just less than the middle 1/8th of the double range. They overlap the normal single-precision floats which also have 8 bit exponents, but exclude the single-precision denormals (exponent-127) and the single-precision NaNs (exponent +127). +/- zero is just a pair of values with both exponent and mantissa 0.

So the non-zero immediate doubles range from
+/- 0x3800,0000,0000,0001 / 5.8774717541114d-39
to +/- 0x47ff,ffff,ffff,ffff / 6.8056473384188d+38

Encoding and decoding

The encoded tagged form has the sign bit moved to the least significant bit, which allows for faster encode/decode because offsetting the exponent can’t overflow into the sign bit and because testing for +/- 0 is an unsigned compare for <= 0xf.

So given the tag is 4, the tagged non-zero bit patterns are
0x0000,0000,0000,001[c(8+4)]
to 0xffff,ffff,ffff,fff[c(8+4)]
and +/- 0d is 0x0000,0000,0000,000[c(8+4)]

Decoding of non-zero values in machine code is performed as follow:

Decoding.png

Encoding of non-zero values in machine code is performed as follow:

Encoding.png

Reading floats in general is fairly easy, the VM checks the class index, if the class index of immediate float is present, then the float is decoded from the oop, if the boxed float class index is present, the float is read from the boxed object.

Each primitive operation (arithmetic, comparison, etc.) has now to be implemented twice, once in both classes, where the first operand is expected to be an instance of the class where it is installed. In Smalltalk float primitive operations succeed if the second operand is one of the 2 float classes or a SmallInteger. It fails for large integers and arbitrary objects, in which case the VM takes a slow path to perform correctly the operation.

At the end of arithmetic operations, the resulting float has to be converted back from the unboxed format to either an immediate float or a boxed float. To do so, the VM checks the exponent of the float against the smallFloatExponentOffset, 896. 896 is 1023 – 127, where 1023 is the mid-point of the 11-bit double precision exponent range, and 127 is the mid-point of the 8-bit SmallDouble exponent range. If the exponent is in range, it can be converted to an immediate float. If not, one needs to check if the float is +/- 0, in which case it can still be converted to an immediate float, else it has to be converted to a boxed float. The code looks like that in Slang:

^exponent > self smallFloatExponentOffset
ifTrue: [exponent <= (255 + self smallFloatExponentOffset)]
ifFalse:
[(rawFloat bitAnd: (1 << self smallFloatMantissaBits - 1)) = 0
ifTrue: [exponent = 0]
ifFalse: [exponent = self smallFloatExponentOffset]]

x86_64 encoding/decoding

To conclude the post, here are the instructions generated in x86_64 to encode immediate floats. I put the instruction so that you can see how to encode efficiently using the theoretical design from the figures above and in addition quick checks for +/- 0.

000020da: rolq $1, %r9 : 49 D1 C1
000020dd: cmpq $0x1, %r9 : 49 83 F9 01
000020e1: jbe .+0xD (0x20f0=+@F0) : 76 0D
000020e3: movq $0x7000000000000000, %r8 : 4D B8 00 00 00 00 00 00 00 70
000020ed: subq %r8, %r9 : 4D 2B C8
000020f0: shlq $0x03, %r9 : 49 C1 E1 03
000020f4: addq $0x4, %r9 : 49 83 C1 04

Decoding is easier:

00002047: movq %rdx, %rax : 48 89 D0
0000204a: shrq $0x03, %rax : 48 C1 E8 03
0000204e: cmpq $0x1, %rax : 48 83 F8 01
00002052: jle .+0xD (0x2061=+@61) : 7E 0D
00002054: movq $0x7000000000000000, %r8 : 4D B8 00 00 00 00 00 00 00 70
0000205e: addq %r8, %rax : 49 03 C0
00002061: rorq $1, %rax : 48 D1 C8
00002064: movq %rax, %xmm0 : 66 48 0F 6E C0

Let me know if you have any question or you want me to expand this post with something else.

Note: Part of the blog post was extracted from the SpurMemoryManager class comment on immediate float, I thank Eliot Miranda and other OpenSmalltalk-VM contributors for writing it.

by Clement Bera at November 09, 2018 07:04 PM

November 07, 2018

PharoWeekly

[Ann] v1.4.0 of Iceberg

Hello!

This week we are releasing the version v1.4.0 of Iceberg.
(https://github.com/pharo-vcs/iceberg/releases/tag/v1.4.0)

This version is available in the latest Pharo 7.

This release fixes a bug introduced in v1.3. It also add new features
in the repository view, add some cleaning and also re-introduce a
feature that was lost.

Thanks to all contributors.

Enjoy!

Full changelog:

Bugfixes

#1068 'There is no associated repository configured.' warning on right
clicking missing repository

Features

#1077 Repository view: Allow to collapse branches/remotes/tags trees
#847 Move tags under remotes in Repository view
#1070 set upstream if missing

Cleanups

#1066 Pharo 7: PackageManifest subclasses should be packaged with "Manifest"
#1015 Replace usages of Glamour in the Github Plugin
#1063 1061-Introduce-iconNamed-in-IceDefinition-and-IceTipModel-and-remove-all-the-terrible-Smalltalk-ui-icons

by Stéphane Ducasse at November 07, 2018 09:10 PM

[ann] Pharo 7.0.0-rc1

Greetings!

I’m announcing today we reach Pharo 7.0.0-rc1!

This is the first step to release a definitive version, and while we will continue integrating bug fixes, API change Pull Requests will be delayed until the open of Pharo 8.0.0 development.
Now, you would wonder what is the ChangeLog of this release… and answer is we still do not have one (btw, we should find a way to automate this).

Anyway… we are very close to release now 🙂

Please download, test, report issues.

by Stéphane Ducasse at November 07, 2018 10:44 AM

November 06, 2018

PharoWeekly

[Ann] release v1.3 of Iceberg

Hello!

This week we are releasing the version v1.3 of Iceberg.
(https://github.com/pharo-vcs/iceberg/releases/tag/v1.3.0
+https://github.com/pharo-vcs/iceberg/releases/tag/v1.3.1)

This version will be available after we merge this PR:

https://github.com/pharo-project/pharo/pull/1951

This release contains some new features such as the support of self
hosted gitlab, integration with github, etc.
It also contains multiple bug fixes, cleanups and enhancements.

On the CI part, Guille made the Appveyor build green! This will
increase the stability of the windows support.

Thanks to all contributors.

Enjoy!

Full changelog:

Features

#1021 Self hosted gitlab support
#1027 Improved new tag dialog to generate semantic versionning tags
#1044 Show a button “View on Github” when creating a PR
#1008 Add “create branch from GitHub issue” option
#1048 Add commands to open remotes on Github
#1010 Add menu entry in extras to force calculate diff

Bug corrections

#975 Metacello asks too many times what to install when there are
conflicting versions
#980 Iceberg should Identify better the packages and the normal files
#982 The Edit Project should have a Warning if it will affect the packages
#986 Iceberg does not realize changes in extended classes
#999 Pulling and pushing to a gitolite server asks password
#984 Conversion to Tonel generates corrupted .properties
#1041 Filter in repository view don’t work with capital letters
#1019 Metacello Integration leaves Monticello leftover repositories
#859 Creating a branch and pushing does not sets the upstream
#1043 Packages starting with lowercase not recognized
#991 Error on right click in the project editon wizard
#775 Reviewing a PR is broken
#1036 Debugger if we try to merge without selecting a branch
#1064 Fix failing tests regarding clean code in Pharo

Enhancements

#988 Iceberg should load the packages in a single MCLoader (This will
make the loads of packages atomic)
#1001 Use “instance creation” instead of “instance-creation” for
method protocol name
#1004 Use displayScaleFactor in UI
#977 Add ToolTip help to the Commands
#1030 Better support for binary files
#1034 SSH passphrase is now hidden

Cleanups

#1018 Iceberg UI relies on deprecated classes from Spec and Commander
#1051 Clean useless huge hierarchy in Github plugin UI

Infrastructure Enhancements

#1023 Fix CI for windows

by Stéphane Ducasse at November 06, 2018 07:00 PM

November 03, 2018

Hernan Morales

Pharo Script of the Day: One-liner identity matrix

Check how easy is to create an identity matrix in Pharo, in just one line: require matrix size from user and print the output:

(Array2D identity: (UIManager default request: 'Enter size of the matrix:') asInteger) asString

by Hernán (noreply@blogger.com) at November 03, 2018 08:08 PM

November 02, 2018

PharoWeekly

[Pharo-dev] [ANN] The STON Specification

Hi,

Since there can never be enough documentation I finally took some time to write a more formal description of STON as a data format.

  https://github.com/svenvc/ston/blob/master/ston-spec.md

The idea is to let this stabilise a bit and to then update the two other documents describing STON, where necessary:

  https://github.com/svenvc/ston/blob/master/ston-paper.md
  https://ci.inria.fr/pharo-contribution/job/EnterprisePharoBook/lastSuccessfulBuild/artifact/book-result/STON/STON.html

Also, the latest changes in STON have to make their way to the Pharo image as well.

  https://github.com/svenvc/ston

All feedback is welcome.

Sven

by Stéphane Ducasse at November 02, 2018 09:58 AM

Torsten Bergmann

Dynamic creation of compiler plugins in Pharo

A new class (merged this week into Pharo 7) called "OCCompilerDynamicASTPlugin" allows for the dynamic creation of compiler plugins


Object compiler
addPlugin:
(OCCompilerDynamicASTPlugin
newFromTransformBlock: [ :ast | (RBParseTreeRewriter replaceLiteral: 42 with: 'meaning of life') executeTree: ast. ast. ]
andPriority: 0
);
evaluate: '42'.
This would give 'meaning of life'

by Torsten (noreply@blogger.com) at November 02, 2018 09:46 AM

October 30, 2018

Hernan Morales

Pharo Script of the Day: Mass image format conversion from PNG to JPEG

You might find useful the following code to convert a whole directory of image in PNG format to JPEG:

(FileSystem disk workingDirectory filesMatching: '*.png') do: [ : pngFile |
pngFile asFileReference binaryReadStreamDo: [ : stream |
PluginBasedJPEGReadWriter
putForm: (PNGReadWriter formFromStream: stream)
onFileNamed: pngFile withoutExtension , 'jpg' ] ]
displayingProgress: 'Converting images to JPG...'.

by Hernán (noreply@blogger.com) at October 30, 2018 04:20 AM

October 26, 2018

Hernan Morales

Pharo Script of the Day: Text analysis using tf-idf

Today's snippet takes a natural language text as input (a.k.a. the Corpus) where each line is considered a different document, and outputs a matrix of term documents with word mappings and frequencies for the given documents. This is also known as tf-idf, a distance metric widely used in information retrieval and provides the relevance or weight of terms in a document.

Why is not this just simple word counting?

If you increase relevance proportionally to word count, then all your query results will have words like "the" as the most relevant in the whole set of documents (or even in a single document), as it is a very common word. So you would need to decrease count for these common words, or increase count for "rare" words to get their relevance. This is where IDF (inverse document frequency) comes into play. With IDF you count documents, so you will assign low score to terms appeared in a lot of documents, then increasing the divider and decreasing relevance.

Finally, Stop words are removed and stemming is performed to reduce words with the same root.

First of all, you can install Moose-Algos (with some needed Hapax classes in a clean Pharo image by evaluating:


Metacello new
configuration: 'MooseAlgos';
smalltalkhubUser: 'Moose' project: 'MooseAlgos';
version: #development;
load.
Gofer it
smalltalkhubUser: 'GustavoSantos' project: 'Hapax';
package: 'Hapax';
package: 'Moose-Hapax-VectorSpace';
load.

Then you can execute the script:

| corpus tdm documents |
corpus := MalCorpus new.
documents := 'Julie loves me more than Linda loves me
Jane likes me more than Julie loves me'.
documents lines doWithIndex: [: doc : index |
corpus
addDocument: index asString
with: (MalTerms new
addString: doc
using: MalCamelcaseScanner;
yourself)].
corpus removeStopwords.
corpus stemAll.
tdm := HapTermDocumentMatrix on: corpus.
tdm.

by Hernán (noreply@blogger.com) at October 26, 2018 02:30 AM

October 25, 2018

Hernan Morales

Pharo Script of the Day: Count lines of code

Lines of code, LOC, SLOC, ELOC... one the simplest and metrics around, and we could find the method with the most LOC in the image with just one line of code (tested in Pharo 6.1):

SystemNavigation default allMethods 
collect: [ : m | m -> m linesOfCode ]
into: (SortedCollection sortBlock: [ : a : b | a value < b value ])

For more advanced software engineering queries have a look to the cool Moose ecosystem

by Hernán (noreply@blogger.com) at October 25, 2018 02:17 AM

October 24, 2018

Hernan Morales

Pharo Script of the Day: SPARQL access to DBPedia


Let's face it, how many times you could have a mix of Natalie Portman with Smalltalk code? :) If you install a little SPARQL wrapper library in Pharo, you could for example access the Natalie's movie list querying DBPedia by writing something like the following code in the SPARQL query language:


DBPediaSearch new
setJsonFormat;
timeout: 5000;
query: 'PREFIX dbpedia-owl: <http://dbpedia.org/ontology/>
SELECT DISTINCT ?filmName WHERE {
?film foaf:name ?filmName .
?film dbpedia-owl:starring ?actress .
?actress foaf:name ?name.
FILTER(contains(?name, "Natalie"))
FILTER(contains(?name, "Portman"))
}';
execute

To actually get only the titles you can use NeoJSON to parse the results:

((((NeoJSONReader fromString: jsonResults) at: #results) at: #bindings) collect: [ : entry | entry at: #filmName ]) 
collect: [ : movie | movie at: #value ]

And this is how results looks like:


by Hernán (noreply@blogger.com) at October 24, 2018 12:11 AM

October 23, 2018

Pierce Ng

Glorp SQLite on Pharo 7

GlorpSQLite works on Pharo 7!

Take a fresh Pharo 7 alpha image; as of yesterday's download that is 5f13ae8. Launch it and run the following snippet in a Playground:

Metacello new
  baseline: 'GlorpSQLite';
  repository: 'github://PierceNg/glorp-sqlite3:pharo7dev';
  load.

Run the Glorp tests in TestRunner. The result should be green, with all 891 tests passed and 12 tests skipped. The database file is sodbxtestu.db in your image directory. Tested on 32- and 64-bit Ubuntu 18.04.

by Pierce Ng at October 23, 2018 09:03 PM

October 22, 2018

Hernan Morales

Pharo Script of the Day: Visualize SVG paths using Roassal

Let's suppose we want to render a SVG shape described in a SVG Path. As SVG is basically XML you can grab (read: parse) the figure coordinates from the SVG path description attribute. For this we can use the XML DOM parser, Roassal and pass just the coordinates found in the "d" attribute of the "path" node, to build more complex shapes, like the following country:

| xmlTree view |
view := RTView new.
xmlTree := (XMLDOMParser onURL: 'https://www.amcharts.com/lib/3/maps/svg/belgiumHigh.svg') parseDocument firstNode.
((xmlTree findElementNamed: 'g')
nodesCollect: [ :node | | elem |
[ elem := (RTSVGPath new
path: (node attributeAt: 'd');
fillColor: Color random;
scale: 0.5) element ]
on: Error
do: [ : ex |
elem ifNotNil: [
elem model: (node attributeAt: 'title').
elem @ RTPopup.
elem ] ]])
reject: #isNil
thenDo: [ : e | view add: e ].
view open

That's basically code extracted from the Territorial library to easily render maps. Have you guessed it yet? Yes, it's Belgium!

by Hernán (noreply@blogger.com) at October 22, 2018 10:15 PM

Pharo Script of the Day: Unzip, the Smalltalk way

Hi everybody. Today a simple but useful script to uncompress a ZIP file in the current image directory. Notice the #ensure: send, Smalltalk provides an very elegant way to evaluate a termination block:

| zipArchive fileRef |
zipArchive := ZipArchive new.
fileRef := 'myFile.zip' asFileReference.
[ zipArchive
readFrom: fileRef fullName;
extractAllTo: FileSystem workingDirectory ]
ensure: [ zipArchive close ].

by Hernán (noreply@blogger.com) at October 22, 2018 01:51 AM

October 21, 2018

PharoWeekly

[Ann] Smacc Book V1.0

The book around Smacc: the compiler-compiler framework is now available in pdf and html.

http://books.pharo.org/booklet-Smacc/

S. Ducasse

 

by Stéphane Ducasse at October 21, 2018 07:01 PM

October 20, 2018

Pierce Ng

Glorp Mapping Existing Schema - Part 2

This is the second post in a short series on the topic. The last post looked at the tables GROUPS and TEAMS in the OpenFootball relational database schema. There is also the table GROUPS_TEAMS, usually known as a link table, which, ahem, "relates" the GROUPS and TEAMS table. GROUPS_TEAMS has the following schema:

CREATE TABLE IF NOT EXISTS "groups_teams" (
  "id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, 
  "group_id" integer NOT NULL, 
  "team_id" integer NOT NULL, 
  "created_at" datetime NOT NULL,
  "updated_at" datetime NOT NULL
);

A row in GROUPS_TEAMS with group_id of XXX and team_id of YYY means that the team represented by team_id YYY is in the group with group_id XXX.

Let's modify the Smalltalk class OFGroup to handle the linkage, by adding the inst-var 'teams' and creating accessors for it.

OFObject subclass: #OFGroup
  instanceVariableNames: 'eventId title pos teams'
  classVariableNames: ''
  package: 'OpenFootball'

Next, modify the mapping for OFGroup in OFDescriptorSystem:

classModelForOFGroup: aClassModel
  self virtualClassModelForOFObject: aClassModel.
  aClassModel newAttributeNamed: #eventId type: Integer.
  aClassModel newAttributeNamed: #title type: String.
  aClassModel newAttributeNamed: #pos type: Integer.
  "Next item is for linking OFGroup with OFTeam."
  aClassModel newAttributeNamed: #teams collectionOf: OFTeam.

descriptorForOFGroup: aDescriptor
  | t | 
  t := self tableNamed: 'GROUPS'.
  aDescriptor table: t.
  self virtualDescriptorForOFObject: aDescriptor with: t.
  (aDescriptor newMapping: DirectMapping)
    from: #eventId
    type: Integer
    to: (t fieldNamed: 'event_id').
  (aDescriptor newMapping: DirectMapping)
    from: #title
    type: String
    to: (t fieldNamed: 'title').
  (aDescriptor newMapping: DirectMapping)
    from: #pos
    type: Integer
    to: (t fieldNamed: 'pos'.
  "Next item is for linking OFGroup with OFTeam."
  (aDescriptor newMapping: ManyToManyMapping)
    attributeName: #teams.

"No change to #tableForGROUPS:."

It is now necessary to add the table GROUPS_TEAMS to OFDescriptorSystem:

tableForGROUPS_TEAMS: aTable
  | gid tid |
  self virtualTableForOFObject: aTable.
  gid := aTable createFieldNamed: 'group_id' type: platform integer.
  aTable addForeignKeyFrom: gid to: ((self tableNamed: 'GROUPS') fieldNamed: 'id').
  tid := aTable createFieldNamed: 'team_id' type: platform integer.
  aTable addForeignKeyFrom: tid to: ((self tableNamed: 'TEAMS') fieldNamed: 'id').

Now let's fetch the OFGroup instances with their linked OFTeam instances.

| vh |
Transcript clear.
OFDatabase dbFileName: 'wc2018.db'
  evaluate: [ :db |
    db session accessor logging: true. "This shows the generated SQL."
    vh := String streamContents: [ :str | 
      (db session read: OFGroup) do: [ :ea | 
        str nextPutAll: ea title; nextPut: Character cr.
        ea teams do: [ :team | 
          str nextPutAll: '- ', team title; nextPut: Character cr ]]]].
vh

The above snippet produces the following output:

Group A
- Egypt
- Russia
- Saudi Arabia
- Uruguay
<some output omitted>
Group H
- Senegal
- Japan
- Poland
- Colombia

In the snippet, logging is enabled, and the SQL generated by Glorp is displayed in the Transcript (with whitespace inserted for readability). What we see is the infamous "N+1 selects problem" in action - the first SELECT fetches the GROUPS rows, then, for each group_id, there is a corresponding SELECT to fetch the TEAMS rows.

SELECT t1.id, t1.created_at, t1.updated_at, t1.event_id, t1.title, t1.pos
 FROM GROUPS t1  an OrderedCollection()

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(1)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(2)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(3)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(4)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(5)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(6)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(7)

SELECT t1.id, t1.created_at, t1.updated_at, t1.key, t1.title
 FROM TEAMS t1, GROUPS_TEAMS t2
 WHERE ((t2.team_id = t1.id) AND (t2.group_id = ?))  an OrderedCollection(8)

Fortunately Glorp is cleverer than this, and provides a way to avoid the N+1 problem, by using the message #alsoFetch:.

| vh |
Transcript clear.
OFDatabase dbFileName: 'wc2018.db'
  evaluate: [ :db |
    | query |
    db session accessor logging: true.
    query := Query read: OFGroup.
    query alsoFetch: [ :ea | ea teams ]. " <== See me. "
    vh := String streamContents: [ :str | 
      (db session execute: query) do: [ :ea | 
        str nextPutAll: ea title; nextPut: Character cr.
        ea teams do: [ :team | 
          str nextPutAll: '- ', team title; nextPut: Character cr ]]]].
vh

Same output as before, but this time the SQL (pretty-printed by hand for readability) is much shorter and properly takes advantage of the SQL language.

SELECT t1.id, t1.created_at, t1.updated_at, t1.event_id, t1.title, t1.pos, 
       t2.id, t2.created_at, t2.updated_at, t2.key, t2.title
FROM GROUPS t1 
INNER JOIN GROUPS_TEAMS t3 ON (t1.id = t3.group_id) 
INNER JOIN TEAMS t2 ON (t3.team_id = t2.id) 
ORDER BY t1.id  an OrderedCollection()

by Pierce Ng at October 20, 2018 10:26 PM

Hernan Morales

Pharo Script of the Day: Massive uncontrolled send and log of unary messages

Want to play and break your VM today? Try this useless saturday script just for fun:

| outStream |
outStream := FileStream newFileNamed: 'unary_sends.txt'.
Smalltalk allClasses
reject: [ : cls | (cls basicCategory = #'Kernel-Processes') or: [ cls = HashedCollection ] ]
thenDo: [ : cls |
cls class methodDictionary
select: [: sel | sel selector isUnary ]
thenCollect: [ : cm |
| result |
result := [ cls perform: cm selector ]
on: Error
do: [ :ex | (ex messageText includes: 'overridden') ifTrue: [ ex pass ] ].
[ result asString ]
on: Error
do: [ : ex2 | result := ex2 messageText ].
outStream nextPutAll: cls asString;
nextPutAll: '>>';
nextPutAll: cm selector asString;
tab;
nextPutAll: result asString; cr. ] ] .
outStream close.

by Hernán (noreply@blogger.com) at October 20, 2018 08:59 PM

October 19, 2018

Hernan Morales

Pharo Script of the Day: A quiz game script to test your Collection wisdom

I want to play a game :) The following script implements an "Is this Sequenceable?" kind of quiz. You are presented with a series of inspectors with method sources in the image, without its class name. And by looking only the source code you have to guess if the method belongs to a SequenceableCollection hierarchy or not. If you miss, you can see the class and its class hierarchy. At the end of the game, you are presenter your score:

| hits n |
hits := 0.
n := 3.
n timesRepeat: [
| mth cls i |
cls := (Collection withAllSubclasses select: #hasMethods) atRandom.
mth := cls methodDict atRandom.
i := GTInspector openOn: mth sourceCode.
((self confirm: 'Method belongs to a Sequenceable Collection?') = (cls isKindOf: SequenceableCollection class))
ifTrue: [ UITheme builder message: 'Good!'. hits := hits + 1 ]
ifFalse: [ UITheme builder message: 'Method class is ' , cls asString , '. Class hierarchy: ' , (cls allSuperclassesExcluding: Object) asArray asString ].
i close ].
UITheme builder message: 'Your score: ' , hits asString , ' / ' , n asString.

What could be done to enhance the script? At first it would be really nice to add an option "Cannot determine with the displayed source"... (TBD) actually there are a lot of possibilities, like asking if it has any Critics, or if could be optimized, etc. Enjoy!

by Hernán (noreply@blogger.com) at October 19, 2018 04:59 AM

October 18, 2018

Pierce Ng

Glorp Mapping Existing Schema - Part 1

Using OpenFootball-Glorp for illustration, this post is the first in a series on mapping an existing normalized database schema and other fun Glorp stuff. As usual, I'm using SQLite for the database.

Consider the tables GROUPS and TEAMS.

CREATE TABLE IF NOT EXISTS "groups" (
  "id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, 
  "event_id" integer NOT NULL, 
  "title" varchar NOT NULL, 
  "pos" integer NOT NULL, 
  "created_at" datetime NOT NULL, 
  "updated_at" datetime NOT NULL
);

CREATE TABLE IF NOT EXISTS "teams" (
  "id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, 
  "key" varchar NOT NULL, 
  "title" varchar NOT NULL, 
-- many other columns omitted for now --
  "created_at" datetime NOT NULL, 
  "updated_at" datetime NOT NULL
);

As it happens, every table in OpenFootball has columns "id", "created_at" and "updated_at", where "id" is that table's primary key. Let's take advantage of Smalltalk's inheritance and class hierarchy to map these columns and tables:

Object subclass: #OFObject
  instanceVariableNames: 'pid createdAt updatedAt'
  classVariableNames: ''
  package: 'OpenFootball'

"Maps to GROUPS."
OFObject subclass: #OFGroup
  instanceVariableNames: 'eventId title pos'
  classVariableNames: ''
  package: 'OpenFootball'

"Maps to TEAMS."
OFObject subclass: #OFTeam
  instanceVariableNames: 'key title'
  classVariableNames: ''
  package: 'OpenFootball'

By convention, the Glorp mapping is encapsulated in the class OFDescriptor, which has these supporting methods:

virtualClassModelForOFObject: aClassModel
  aClassModel newAttributeNamed: #pid type: Integer.
  aClassModel newAttributeNamed: #createdAt type: DateAndTime.
  aClassModel newAttributeNamed: #updatedAt type: DateAndTime.

virtualDescriptorForOFObject: aDescriptor with: aTable
  (aDescriptor newMapping: DirectMapping)
    from: #pid
    to: (aTable fieldNamed: 'id'). "This is the primary key mapping."
  (aDescriptor newMapping: DirectMapping)
    from: #createdAt
    type: DateAndTime
    to: (aTable fieldNamed: 'created_at').
  (aDescriptor newMapping: DirectMapping)
    from: #updatedAt
    type: DateAndTime
    to: (aTable fieldNamed: 'updated_at').

virtualTableForOFObject: aTable
  (aTable createFieldNamed: 'id' type: platform serial) bePrimaryKey.
  aTable createFieldNamed: 'created_at' type: platform datetime.
  aTable createFieldNamed: 'updated_at' type: platform datetime.

The mapping for OFGroup is as follows:

classModelForOFGroup: aClassModel
  self virtualClassModelForOFObject: aClassModel.
  aClassModel newAttributeNamed: #eventId type: Integer.
  aClassModel newAttributeNamed: #title type: String.
  aClassModel newAttributeNamed: #pos type: Integer.

descriptorForOFGroup: aDescriptor
  | t | 
  t := self tableNamed: 'GROUPS'.
  aDescriptor table: t.
  self virtualDescriptorForOFObject: aDescriptor with: t.
  (aDescriptor newMapping: DirectMapping)
    from: #eventId
    type: Integer
    to: (t fieldNamed: 'event_id').
  (aDescriptor newMapping: DirectMapping)
    from: #title
    type: String
    to: (t fieldNamed: 'title').
  (aDescriptor newMapping: DirectMapping)
    from: #pos
    type: Integer
    to: (t fieldNamed: 'pos'.

tableForGROUPS: aTable
  self virtualTableForOFObject: aTable.
  aTable createFieldNamed: 'event_id' type: platform integer.
  aTable createFieldNamed: 'title' type: platform varchar.
  aTable createFieldNamed: 'pos' type: platform integer.

The mapping for OFTeam is similar and I've not shown it here for brevity.

To round out the scene setting, OFDatabase, the "database interface" class, has class-side convenience methods to run snippets like so:

OFDatabase 
  dbFileName: 'wc2018.db'
  evaluate: [ :db |
    db session read: OFGroup ]

To be continued...

by Pierce Ng at October 18, 2018 09:57 PM

PharoWeekly

[ann] MemCached Pharo client

Hi,

I copied the (Pharo) Memcached client to https://github.com/svenvc/memcached where is lives in Tonel format with a Baseline and a working Travis CI build against an actual memcached server in the worker.

More about memcached

- https://en.wikipedia.org/wiki/Memcached
- http://memcached.org

Acknowledgements

The original project can be found at http://www.squeaksource.com/memcached.html

As far as I can see it was written by Philippe Marschall and Ramon Leon. I ported the codebase to Pharo. This repository is a recent copy with some cleanups.

I intend to maintain this as I need it myself. The codebase should still maintain its original portability (minus the meta info, possibly).

Sven

by Stéphane Ducasse at October 18, 2018 07:32 PM

October 17, 2018

PharoWeekly

Internship around 3D/Pharo

https://jobs.thalesgroup.com/job/brest/stage-bac-5-prototypage-ihm-realite-virtuelle-augmentee-h-f/1766/9675118

With the people that did a nice virtual

and event touch with Pharo

S.

by Stéphane Ducasse at October 17, 2018 07:02 PM

October 16, 2018

Hernan Morales

Pharo Script of the Day: Find your IP address

I' back :)

Today let's update the PSotD blog with a script to find your IP address using Zinc HTTP Components. Credits also to Sven Van Caekenberghe which helped me to figure out why Zn was getting a 403

ZnClient new
systemPolicy;
beOneShot;
url: 'http://ifconfig.me/ip';
accept: ZnMimeType textPlain;
headerAt: 'User-Agent' put: 'curl/7.54.0';
timeout: 6000;
get.

by Hernán (noreply@blogger.com) at October 16, 2018 08:50 PM

PharoWeekly

[Ann] Nano Pi Neo + MCP9808 :)

Nano Pi Neo running PharoThings and controlling the GPIOs to take the temperature out of the MCP9808 sensor and turn on/off a LED

 

Great job allex.

by Stéphane Ducasse at October 16, 2018 06:16 PM

PharoThings HC-SR04 ultrasonic sensor

by Stéphane Ducasse at October 16, 2018 10:04 AM

October 15, 2018

Torsten Bergmann

Squeak 5.2. released

Squeak 5.2. is released

by Torsten (noreply@blogger.com) at October 15, 2018 08:14 AM

October 14, 2018

PharoWeekly

[Ann] New roassal video

by Stéphane Ducasse at October 14, 2018 07:31 PM