Training Samples

# ...[33999 | 4308.76] loss=2.45 avg=1.86
# ======== SAMPLE 1 ========
# 11|But I have had a day that was gone when I saw you in a dream,
# 1322|A year ago, a year ago.... I'm going back
# 1322|Oh, the years are many and the years begin!
# 1322|You and I have traveled far and wide,
# 1322|You and I have wandered far and wide
# 1322|Through the land I love so well,
# 1322|Where the beautiful land of long ago
# 1322|Died and ebbed and glowed,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|You and we have traveled far and wide
# 1322|O'er the land we love so well,
# 1322|In the little green land of long ago,
# 1322|In the little green land of long ago.
# 1322|You and I have traveled far and wide
# 1322|Through the land of long ago;
# 1322|And the years have slowly swept us on
# 1322|Till the dust has left the sod,
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|And we all must rue the little day
# 1322|When we all are gone from God....
# 1322|I know a little garden very old,
# 1322|Built in a nook near the western sea,
# 1322|With doors and windows closely locked and barred
# 1322|And steps that made no sound but the world outside,
# 1322|And a wall of white-walled flowers all in a row
# 1322|Of violet bloom that just began to blow,
# 1322|And a dial that looked like a golden globe,
# 1322|Cool, square, and green, with the leaves and the grass,
# 1322|And a dial that looked like a needle of bronze;
# 1322|And I wonder if these petals are lost to-day,
# 1322|These petals will be forgotten to-morrow.
# 1322|"A rose would be a rose,"
# 1322|I wonder much at it;
# 1322|It does not matter what or who,
# 1322|So very, very much it matters:
# 1322|It is so nice to think
# 1322|The world is such a place for petals,
# 1322|And so much like a pink.
# 1322|"A purple fox-hawk lived down by the sea
# 1322|And danced and slept in the white-walled flowers,
# 1322|And I thought about a tree,
# 1322|A rose that dropped as an apple falls
# 1322|Into the water, and fell in showers."
# 1322|And I wondered what people said,
# 1322|And wherefore took the trouble,
# 1322|And wherefore came my little red bird
# 1322|To look at me and wave its wings
# 1322|For joy of self and for its red,
# 1322|And why I cannot tell;
# 1322|For it is so nice to think
# 1322|The world is such a place for petals.
# 1322|And so I sat and played,
# 1322|And the old clock set the chimney-top
# 1322|To ring time to itself and stop
# 1322|The time with a slow, frost-ridden drone,
# 1322|And a rustle as of bees.
# 1322|The house was old, and had grown lean
# 1322|As if it were but a leaf,
# 1322|But I found there was only one room
# 1322|And I slept on my stem at the end of the porch
# 1322|Like a child asleep.
# 1322|The door was old, but very young,
# 1322|And very, very young;
# 1322|And a weary smile on its satin lips
# 1322|Hung like a crown upon the night,
# 1322|And the roses crept into the room
# 1322|By the little red door in the wall,
# 1322|And I woke, and felt as a king is,
# 1322|And thought of all the trouble and pain
# 1322|Of a little life and a little love
# 1322|When only a boy is born.
# 1322|For my heart was old, and restless with the years,
# 1322|And my face with no trace of the things I had
# 1322|And no remembrance when I was twenty-five,
# 1322|And
#
# [34000 | 4318.77] loss=2.15 avg=1.86
# ...
# [47998 | 11504.75] loss=1.91 avg=1.84
# [47999 | 11505.25] loss=2.37 avg=1.85
# ======== SAMPLE 1 ========
# |So when the sun has touched the hill
# 30599|And made the earth a little heave
# 30599|Of gold-dust on the firmament.
# 30599|The sea is a molten pearl
# 30599|Silvering the yellow moon.
# 30599|Gold is the poppies of purple plush,
# 30599|Which make a star within a pool
# 30599|Of purple, blue, and green and blue.
# 30599|The sea is a molten pearl,
# 30599|Whose petals burn as light
# 30599|In the rose-yarn of the sea.
# 30599|The gold star beckons, sparkles, shines.
# 30599|The water gleams and glows.
# 30599|The poppies turn to a golden rim,
# 30599|And a thousand petals, one by one,
# 30599|Come dancing through the purple waves
# 30599|Of purple.
# 30599|My soul goes up to the sun.
# 30599|The sun, all gold, is gone.
# 30599|The sun is gone, the sun
# 30599|Is not more gold than my soul.
# 30599|What is the rainbow, my soul?
# 30599|The rain is falling in the tree
# 30599|In dewdrops falling, that are wet
# 30599|With dewdrops falling.
# 30599|I have heard music in the woods
# 30599|Under a great sky.
# 30599|Their notes, on a hundred harps,
# 30599|Dance by a liquid, falling star.
# 30599|The song of the whole universe
# 30599|Rings like a loon's tune.
# 30599|The rain is falling, my soul,
# 30599|In the tree-tops, the rain is falling,
# 30599|And the rain is not more musical.
# 30599|The trees, like great globed fruit in a garden of heaven,
# 30599|Are as full of little shining blossoms
# 30599|As the face of a child of ten minutes.
# 30599|The wind is playing a soft tune
# 30599|Like the silver notes of a bell.
# 30599|The grass is a dance for a child,
# 30599|And the sun is going down.
# 30599|The rain is playing a soft tune
# 30599|Like the golden notes of a bell.
# 30599|The rain is playing asleep in the meadows
# 30599|Like a sea of dreams,
# 30599|And the wind is playing a soft tune.
# 30599|The leaves are wet with the tears:
# 30599|There is only the wind that comes.
# 30599|The leaves are wet with the tears.
# 30599|The trees have bent to the wind,
# 30599|Like heavy blossoms, and nothing stays.
# 30599|The earth is flooded with tears
# 30599|Like small white stars in the sky.
# 30599|The wind's song is marching by
# 30599|Like a song in a wind from a string.
# 30599|It comes from nothing, it comes not from the years;
# 30599|An atom of dust and a tide,
# 30599|The ceaseless rain falls heavily over the world,
# 30599|The summer flowers are red
# 30599|And one like a gold cup, in a crimson cup,
# 30599|Flows out from nothing, and goes
# 30599|Without speech or motion nor sound.
# 30599|The yellow flowers all slip,
# 30599|All their leaves are wet,
# 30599|And their crimson petals are wet.
# 30599|The rain has driven the sun outside,
# 30599|The wind has driven the rain outside,
# 30599|The moon is going out and in,
# 30599|With the stars on the roof like snow.
# 30599|The rain has driven the rain outside,
# 30599|The moon is driving the clouds in the air
# 30599|Like white, wool flakes in a snow.
# 30599|The trees are wet with the tears,
# 30599|They weep in their night-black tatters,
# 30599|They moan like mortals who lost their way
# 30599|Because they were glad of the rain.
# 30599|The rain is falling, my soul,
# 30599|It has torn away the flowers,
# 30599|They are wet with the tears of the sun.
# 30599|In the black ooze and the snow,
# 30599|The cold winds, shuddering, are blowing,
# 30599|The leaves are blown like blown corn,
# 30599|There is only the wind that comes.
# 30599|Black and long!  The night is
#
# [48000 | 11515.52] loss=2.14 avg=1.85
#
# ...
# [166998 | 5752.38] loss=2.17 avg=1.52
# [166999 | 5752.88] loss=2.10 avg=1.53
# ======== SAMPLE 1 ========
# 26|His golden radiance is no more."
# 16452|"I shall not think of men in Argos more
# 16452|Than they are now, who many a bloody deed
# 16452|Wrought on the Greeks, nor yet of Ilium's king
# 16452|In arts and arts like these can speak the rest.
# 16452|But they--their kings--the Trojans and their sons
# 16452|Have fallen. Their deaths, the Grecians and their friends
# 16452|Have fallen in battle, from whom little hope
# 16452|To escape the battle, but the steadfast hearts
# 16452|Of heroes and of Trojans have become
# 16452|Inglorious still. The immemorial strife
# 16452|Shall rise for ever in a glorious day,
# 16452|When wars are waged between us and the Greeks.
# 16452|The battle shall be theirs, the mirth, the song;
# 16452|The mirth which all the Olympian people share,
# 16452|Shall bless the younger warriors with a joy
# 16452|So great, so glorious, and a greater fame,
# 16452|That all the Greeks shall learn, that in the van
# 16452|Ye stand yourselves, and they will praise your deeds.
# 16452|But I beseech you, if indeed by mine
# 16452|Unknown dishonour you be wrested hence,
# 16452|That with your lusts, illustrious and august,
# 16452|All others ye may vanquish. Now, my friend,
# 16452|Behold this prize to crown your father's pride.
# 16452|He said, and shaking both his palms, assent
# 16452|That I should also wish it. Thou art brave;
# 16452|Thou know'st how Menoetiades the swift
# 16452|Was dragged, of Hector and the fierce compeers
# 16452|And Phrygian warriors. So, we will dispatch
# 16452|Your bodies, then, yourselves to burn the ships
# 16452|In sacrifice; with torches and with bells
# 16452|To burn them, and with oxen to replace
# 16452|Your gallant friends for ever. But I wish
# 16452|That no man living has so long endured
# 16452|The onset of his foes, as I have power
# 16452|To burn or storm; for mighty Hector erst
# 16452|Was slain, and now returns his safe return
# 16452|To the Thesprotians, and each other's wives
# 16452|And tender children, and all other babes
# 16452|Assemble round me now, for ye have more
# 16452|To suffer than they know. Go then--the rest
# 16452|Will bear you safely; if ye dare to use
# 16452|The Grecians with your ships, leave not the flames
# 16452|Or fire, or slay yourselves and leave the rest,
# 16452|Ye may, at least, escape, but Hector's hands
# 16452|Will set you free; the vultures on the hills
# 16452|Held forth, and I will pierce those hollow shades
# 16452|Adown the trench, and ye shall find them there
# 16452|Scaped unaware. If, then, the Gods above
# 16452|Should bring thee to the fane of Peleus' son,
# 16452|In his despite, thou wouldst be left alone,
# 16452|Beside the yoke unwearied, there to stand,
# 16452|And from the Trojans scatter all thy dead.
# 16452|So spake the son of Atreus, and retired.
# 16452|Then, leaving Hector's body in the tent
# 16452|Of Periphas, King Agamemnon seized,
# 16452|And in his royal hand his spear adorned
# 16452|With silver, and with gold, studded thick all o'er
# 16452|With glittering tin, the weapon of Mars thus toil'd.
# 16452|Brave as he is, and of unwearied arm,
# 16452|So shall be thy revenge, if thou shalt make
# 16452|Fierce contest for the Danaï. But fly
# 16452|With shout and clamour fill'd throughout the host.
# 16452|So shall ye perish, slain by Hector's hands,
# 16452|Each living soul, and, flying into fire,
# 16452|Sink under Troy's wide gulfs. Be this the work
# 16452|Of your brave sires, and all the parents' hearts
# 16452|Shall prosper, one and all. But ye in haste
# 16452|(For ye are valiant) shall
#
# [167000 | 5762.79] loss=1.32 avg=1.53
# [167001 | 5763.29] loss=1.60 avg=1.53

One training sample is worth remarking on:

# [49999 | 12526.44] loss=1.83 avg=1.77
# ======== SAMPLE 1 ========
# ._
# 19221|The curfew tolls the knell of parting day,
# 19221|The lowing herd winds slowly o'er the lea,
# 19221|The ploughman homeward plods his weary way,
# 19221|And leaves the world to darkness and to me.
# 19221|Now fades the glimmering landscape on the sight,
# 19221|And all the air a solemn stillness holds,
# 19221|Save where the beetle wheels his droning flight,
# 19221|And drowsy tinklings lull the distant folds.
# 19221|Save that from yonder ivy-mantled tower
# 19221|The moping owl does to the moon complain
# 19221|Of such as, wandering near her secret bower,
# 19221|Molest her ancient solitary reign.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some mirage she lies,
# 19221|Majestic yet majestic, and of seems
# 19221|The image of the unconquerable skies.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, in some majestic cell,
# 19221|Where lordly seraphs strew their balmy dreams
# 19221|On the still night, or in their golden shell.
# 19221|There, in the calm of some Platonic dream,
# 19221|Sits she, and views the unclouded moon arise
# 19221|Like a fair lady full of realms divine;
# 19221|And, all at once, a stony face and bright
# 19221|Glittering in moonlight, like the noon-tints of a night.
# 19221|Methinks I see her in her blissful dreams:
# 19221|--Or, fancy-like, among the moonlight pale,
# 19221|A heavenly lark twitters with every gleam
# 19221|Of the rapt rapt beauty that she craves to hail.
# 19221|Or in the moonlight gleams she seems to see
# 19221|A virgin glory, and a power divine,
# 19221|And every motion of the looking-glass
# 19221|Is full of love and divine gentleness.
# 19221|Methinks I hear her in the twilight dim:
# 19221|--Or is it fancy-free, and musical,
# 19221|And full of music? Or some holy hymn
# 19221|Over some water-lily's long-lost lute?
# 19221|O no! it is not fancy-free, I see:
# 19221|Nay! fain would I thy spirit-hearts invest,
# 19221|And hear the paean of these dreamy rhymes,
# 19221|Might I but speak of my departed rhymes,
# 19221|And thou wouldst sing of my empoet-swans.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream,
# 19221|And like a breeze sings the blue mists of night.
# 19221|O no! it is not fancy-free, I see:
# 19221|Even Poetry murmurs in a pensive dream.
# 19221|How soft the zephyrs, how the languid hours,
# 19221|As in the noonday shade their limbs recline!
# 19221|How mellow streams the mossy banks assume!
# 19221|How solemn woods, low-chiselled, sumptuous green,
# 19221|Nod to the cadence of their hoarse refrain,
# 19221|As if the deep-toned muses' melodies
# 19221|With half-averted plaints and half-averted plumes
# 19221|In some majestic temple's quietness
# 19221|Had to the silver twilight slowly come.
# 19221|How solemn woods, low-couched, around thee lie:
# 19221|--Or is it fancy-free, and melody
# 19221|That makes the dull night long in worship held?
# 19221|Methinks I hear the harp's harmonious sound
# 19221|In some dim wood, when the deep shadows fall;
# 19221|And the low wind, like one that listens, makes
# 19221|In the still woods the harmony of all.
# 19221|Or in the moon's pale beam, on some hoar rock,
# 19221|Lonely and spectral, mourns her feeble woe;
# 19221|And as the slow waves roll, and, ebbing, break
# 19221|In music
#
# [50000 | 12536.67] loss=1.36 avg=1.76

The rhyming in this sample is so good as to be suspicious. It might also sound familiar—because many of these lines are being copied from Thomas Gray’s Elegy Written in a Country Churchyard, which opens:

The curfew tolls the knell of parting day,
The lowing herd wind slowly o’er the lea
The ploughman homeward plods his weary way,
And leaves the world to darkness and to me.
Now fades the glimm’ring landscape on the sight,
And all the air a solemn stillness holds,
Save where the beetle wheels his droning flight,
And drowsy tinklings lull the distant folds;
Save that from yonder ivy-mantled tow’r
The moping owl does to the moon complain
Of such, as wand’ring near her secret bow’r,
Molest her ancient solitary reign.

Some spelling differences aside, this intro is almost entirely copied from the 8 copies of Gray’s poem in the corpus; this extensive copying is not something I spotted in the GPT-2-poetry samples I looked at, suggesting that the scaffolding of the metadata did indeed help with learning.

Also interestingly, the copying only goes so far, as immediately after the final line about the owl, where Gray continues:

…Beneath those rugged elms, that yew-tree’s shade,
Where heaves the turf in many a mould’ring heap,
Each in his narrow cell for ever laid,
The rude Forefathers of the hamlet sleep …

GPT-2-poetry-prefix instead continues:

…Methinks I see her in her blissful dreams:
—Or, fancy-like, in some mirage she lies,
Majestic yet majestic, and of seems
The image of the unconquerable skies.
Methinks I see her in her blissful dreams:
—Or, fancy-like, in some majestic cell,
Where lordly seraphs strew their balmy dreams
On the still night, or in their golden shell.
There, in the calm of some Platonic dream,
Sits she, and views the unclouded moon arise
Like a fair lady full of realms divine;
And, all at once, a stony face and bright
Glittering in moonlight, like the noon-tints of a night. …

That is, it focuses on the female figure of the Moon in a way more ode-like than elegiac. These lines also do not seem to be extracted from the rest of Elegy either, as words like “bliss” or “mirage” or “dream” or “seraph” or “Platonic” do not appear in it. Some of the phrases like “blissful dreams” do appear in the rest of the corpus, but others like “some mirage” or “mirage she” do not. Nevertheless, the style is consistent throughout the entire sample and the quality is good, suggesting that while GPT-2-poetry-prefix has managed to memorize to a limited extent, it is nevertheless fully capable of generating good original text.

An additional example of memorization has been spotted; sample #17 in the 1,000 unconditional samples is almost entirely a memorized copy of Percy Bysshe Shelley’s “To a Skylark”:

======================================== SAMPLE 17 ========================================
 of our mortal life, and made it known
32373|To be man's also,--as to make us one;
32373|A thing apart, yet of an individual
32373|In frailty and high elemental own.
32373|Hail to thee, blithe Spirit!
32373|Bird thou never wert,
32373|That from heaven, or near it
32373|Pourest thy full heart
32373|In profuse strains of unpremeditated art.
32373|Higher still and higher
32373|From the earth thou springest
32373|Like a cloud of fire;
32373|The blue deep thou wingest,
32373|And singing still dost soar, and soaring ever singest.
32373|In the golden lightning
32373|Of the sunken sun
32373|O'er which clouds are brightening,
32373|Thou dost float and run,
32373|Like an unbodied joy whose race is just begun.
32373|The pale purple even
32373|Melts around thy flight;
32373|Like a star of heaven,
32373|In the broad daylight
32373|Thou art unseen, but yet I hear the music of thy singing.
32373|Keen as are the arrows
32373|Of that silver sphere,
32373|Whose intense lamp narrows
32373|In the white dawn clear,
32373|Until we hardly see, we feel that it is there.
32373|All the earth and air
32373|With thy voice is loud,
32373|As, when night is bare,
32373|From one lonely cloud
32373|The moon rains out her beams, and heaven is overflowed.
32373|What thou art we know not;
32373|What is most like thee?
32373|From rainbow clouds there flow not
32373|Drops so bright to see,
32373|As from thy presence showers a rain of melody.
32373|Like a poet hidden
32373|In the light of thought,
32373|Singing hymns unbidden,
32373|Till the world is wrought
32373|To sympathy with hopes and fears it heeded not;
32373|Like a high-born maiden
32373|In a palace tower,
32373|Soothing her love-laden
32373|Soul in secret hour
32373|With music sweet as love, which overflows her bower;
32373|Like a glow-worm golden,
32373|In the silver dew,
32373|Scattering unbeholden
32373|Its aerial hue
32373|Among the flowers and grass, which screen it from the view;
32373|Like a rose embowered
32373|In its own green leaves,
32373|By warm winds deflowered,
32373|Till the scent it gives
32373|Makes faint with too much sweet these heavy-winged thieves.
32373|Sound of vernal showers
32373|On the twinkling grass,
32373|Rain-awakened flowers,
32373|All that ever was
32373|Joyous and fresh and clear thy music doth surpass.
32373|Teach us, sprite or bird,
32373|What sweet thoughts are thine:
32373|I have never heard
32373|Praise of love or wine
32373|That panted forth a flood of rapture so divine.
32373|Chorus hymeneal,
32373|Or triumphant chant,
32373|Match'd with thine, would be all
32373|But an empty vaunt--
32373|A thing wherein we feel there is some hidden want.
32373|What objects are the fountains
32373|Of thy happy strain?
32373|What fields, or waves, or mountains?
32373|What shapes of sky or plain?
32373|What love of thine own kind? What ignorance of pain?
32373|With thy clear keen joyance
32373|Languor cannot be:
32373|Shadow of annoyance
32373|Never came near thee:
32373|Thou lovest; but ne'er knew love's sad satiety.
32373|Waking or asleep,
32373|Thou of death must deem
32373|Things more true and deep
32373|Than we mortals dream,
32373|Or how could thy notes flow in such a crystal stream?
32373|We look before and after,
32373|And pine for what

The 87 lines beginning with “Hail to thee, blithe Spirit!” are all Shelley (with perhaps slight spelling differences), much surpassing the memorization for Thomas Gray. Considering the top-k sampling method, it’s amazing that the sample could so exactly follow “To A Skylark”. It turns out there are ~12 copies of the poem in the PG corpus (it’s a popular poem), so in retrospect some degree of memorization is not surprising, but that’s still a lot of memorization. The 4 lines beforehand don’t appear to be copied from another Shelley poem, making it even more amazing. It’s a pity that that sample did not continue further because one wonders whether it could have repeated the entire poem and what it would’ve done when the original poem ended.

Unconditional Samples

For both GPT-2s, I generated 1000 samples as follows:

python src/generate_unconditional_samples.py --top_k 40 --temperature 0.9 --nsamples 1000 --seed 0 \
 --model_name 2019-03-06-gwern-gpt2-poetry-prefix-projectgutenberg-network-224474
# ======================================== SAMPLE 1 ========================================
# |But I shall tell thee of the glorious days
# 1008|Of that old strife, wherein the truth of it
# 1008|Atoned, though at the most points of the pulp
# 1008|Be passed through: and about the torment, which
# 1008|A Hermit in his youth thoughte, yet made
# 1008|Still worse by his words, where he said, 'Peace!
# 1008|Keep silence here; because in any place
# 1008|I heard of charity the while I dwelt,
# 1008|And of her tears made e'en my lot a sting:
# 1008|And, but for those, in truth, to heaven were found
# 1008|The spirit of him offered by the sea,
# 1008|So that, out struggling, he made no denial.
# 1008|But the proud spirit, soon as he was silent,
# 1008|Embraced me, even in bidding me speak therefore,
# 1008|And with such gentle voice and such great front
# 1008|Came up unto my Saviour, that he made
# 1008|Full many a sign, how that all virtue he
# 1008|Would at his own discretion hear and mark;
# 1008|Then saw he the chief flower of love delight
# 1008|Within the living light, and made such pact,
# 1008|That it would make bliss vouchsafe to him.
# 1008|Paradiso: Canto XXIX
# 1008|The day, that most of us had sojourning
# 1008|'Mong the sweet spirits, from the highest shaft
# 1008|Of their kind master had not looked on them,
# 1008|Into the eyes forth from the bark they came;
# 1008|Two palms it bore of one and of the letters,
# 1008|And the other two with wings outspread.
# 1008|Not in beatitude do I behold them,
# 1008|But in and out of hope they uttered a
# 1008|Heaven-toned word of comfort, so that reached
# 1008|The upper light, which in attention hung
# 1008|With the low shores, that to the nether shores
# 1008|Were as the utmost thread 'to the high point.
# 1008|And that unceasing after-effiguring
# 1008|Of the eternal Sire, upon the which
# 1008|One live soul only drew its breath mysteriously,
# 1008|Up to the point, so pointed to the source
# 1008|Of what it had been, I never it knew,
# 1008|And hence never would have recognized.
# 1008|What verdure of an undivided heart
# 1008|Resolved, I say; and he complained e'en thus:
# 1008|"What from thy soul unto the Good I send
# 1008|I ever pray to, and by grace of that
# 1008|Pointing myself I pray thee to the world,
# 1008|To point thee the great mystery of love,
# 1008|From this, the bottom to the source of all
# 1008|Concerning thee; and not by its green leaves
# 1008|Of science so unsullied was the thought,
# 1008|As a small-handled cup, acquired by men.
# 1008|The mind's eye, taking from the mortal world
# 1008|All that it asks of bar or of the gold,
# 1008|With the same fury burns as it was wont;
# 1008|Now it may be by lantern or by shining,
# 1008|Since both thy and my love has made me its."
# 1008|The Almighty Father in his thunder made
# 1008|Resenting, and all round about Him round
# 1008|Went down his smitten steps, so that the air
# 1008|Impregnate came not from his visitations,
# 1008|Setting a day of darkness on all sides.
# 1008|Therefore mine eyes I lifted to the ground,
# 1008|And I beheld a river by the ice
# 1008|Chained up and flowing back along the ice,
# 1008|And suddenly before my feet it melted;
# 1008|And what it now behoves me to retrace
# 1008|The cause I had of it in heart I felt.
# 1008|As the Sicilian bull, that rightfully
# 1008|His cries first echoed in the mountains,
# 1008|Did so rebellow, with the sound of which
# 1008|It made my very blood to quicken well,
# 1008|The dolorous accents which envenom'd me,
# 1008|Forthwith I hasten'd unto where reply
# 1008|Was made: "O Ro! Brunhild"
# ======================================== SAMPLE 2 ========================================
# |Hear the tale that the funeral chant is telling,
# 2491|For the sorrows of other's children that dwell
# 2491|Like sweet flowers upon the wold?
# 2491|'Tis the tale of a life which is fled and gone,
# 2491|And the star of a hope which shone
# 2491|Bright above it, though dark may it be,
# 2491|For the hopes of a brighter day are fled
# 2491|And the joys of a happier lot?
# 2491|'Tis the tale of a life with the weary and sad,
# 2491|Where sorrows begin and rest.
# 2491|For only a song can the widow's soul glad
# 2491|Who sits musing 'mid shadows drear.
# 2491|And only a music, sad with its sighs,
# 2491|Till sad to the soul as death draws near
# 2491|As life on her fragile bark!
# 2491|I hear their voices faint in my slumbrous sleep,
# 2491|The music of lives that seem less real
# 2491|Than phantoms are dream-bound in duty's mystic keep,
# 2491|With music that seems to be more real
# 2491|Than phantoms are dream-bound in duty's mystic keep
# 2491|For souls that sin may not see!
# 2491|All round about us seems, in every place,
# 2491|As far off as the eyes of kith and kin,
# 2491|The ever-tremulous busy world's harmonious race,
# 2491|And I hear the mighty ocean tides,
# 2491|Feeling their strength, their might, their rhythmic din,
# 2491|Are calling me all into one wide choral face,
# 2491|And I hear the infinite singing of the winds,
# 2491|That seem to make me simply live!...
# 2491|The world seems a world that is full of sound and motion;
# 2491|A world of beauty and of music, where it lies;
# 2491|Yet all that is and has for me seems one more treasure
# 2491|Than all the world dreams leave in the skies.
# 2491|I hear the mighty tides of life,
# 2491|They're crying to me,
# 2491|They rise and sink in a restless strife
# 2491|Of endless song.
# 2491|Yet every stroke of sorrow's sword
# 2491|Comes surely from afar,
# 2491|That is true peace which is hard on board
# 2491|Though oceans be dark and terrors war.
# 2491|I hear the myriad singing words
# 2491|Of ocean's depths,
# 2491|They come like a song of broken birds,
# 2491|The music floats on the air and stirs
# 2491|My life to bear its measure in calms
# 2491|Of perfect peace, and it is good,
# 2491|But all is false peace only.
# 2491|When first I heard the autumn rain
# 2491|Sink down the hollows on the plain,
# 2491|I held it very near,
# 2491|And as I spoke to March again
# 2491|I felt the long, slow throbbing rain
# 2491|Creep from the earth in sudden flight
# 2491|Through all the veins of earth again,
# 2491|And in the sunlit, silent night
# 2491|The world grew far forlorn.
# 2491|And April came with rushing rains,
# 2491|And leaves about the naked lanes.
# 2491|I saw again the August noon
# 2491|Roll round the world in blazing heaps.
# 2491|And in the sunlight and the dark
# 2491|A thousand germs their pageant crush.
# 2491|And from the earth the maples bloom
# 2491|In odors of the breath of bloom
# 2491|And from the meadows and the hills
# 2491|The rosy clouds drop down their spilled spilled spilled spilled
# 2491|And drunken with the rain it kills.
# 2491|And soon above the hills shall crash
# 2491|The thunder of rain-wings,
# 2491|And all the naked trees and shrubs
# 2491|Shall lie, like naked, naked blades.
# 2491|Out on the hills there shall be rain,
# 2491|And the maples down the windy lane
# 2491|Shall bleed, and flowers shall weep again
# 2491|Through the weary hours of rain.
# 2491|They shall lie where the maples lie
# 2491|Deep in their bosoms, cold and numb,
# 2491|Each with its wound on either arm,
# ...

Download links again:

• GPT-2-poetry, 1,000 unconditional generation samples (3.6MB)

• GPT-2-poetry-prefix, 1,000 unconditional samples (3.2MB)

Some fun passages I noticed in the first 100 unconditional samples:

======================================== SAMPLE 2 ========================================
|Hear the tale that the funeral chant is telling,
2491|For the sorrows of other's children that dwell
2491|Like sweet flowers upon the wold?
2491|'Tis the tale of a life which is fled and gone,
2491|And the star of a hope which shone
2491|Bright above it, though dark may it be,
2491|For the hopes of a brighter day are fled
2491|And the joys of a happier lot?
2491|'Tis the tale of a life with the weary and sad,
2491|Where sorrows begin and rest.
2491|For only a song can the widow's soul glad
2491|Who sits musing 'mid shadows drear.
2491|And only a music, sad with its sighs,
2491|Till sad to the soul as death draws near
2491|As life on her fragile bark!
...
## Sample 3:
...
37804|The white-petalled white fox
37804|Opens himself to coolness
37804|In the late evening.
37804|But when the last child started
37804|The white fox to his feet flew,
37804|And the old fox was master
37804|Of all the magic heathen.
37804|Till when the faint huntsman
37804|Had snuffed the fragrant water
37804|Over his plump ears and skin,
37804|In the old way he knew not
37804|Till morn had almost shone;
37804|And then the fox came slowly
37804|And left the place unguessed;
37804|The white fox was not master,
37804|Although he had been master,
37804|Although he had been servant
37804|And now he could be master
37804|Of all the magic powers
37804|That keep the place enchanted
37804|In the wide earth and water.
...
## Sample 9:
...
36661|And the morn breaks, and, all the day,
36661|Red-clover'd birds with silver bill
36661|Flutter from tree to tree in flower,
36661|A quivering dew, a wind that wafts
36661|To haunts among the ancient woods.
36661|The golden-crested ilex, here
36661|Doth vine her purple cup; the deer,
36661|The wild-goose; and, in troops, the sheep,
36661|The goat, the sylvan-haunted elm,
36661|And the green-faced oft-gadding pine
36661|Blossom with purple.
36661|The lark soars up,
36661|And the hare loud answer make!
36661|Doves, willows, dunes, aslant the lake;
36661|Pair after pike sounds warbling;
36661|The reeds a triumph!
...
## Sample 14:
...
37452|I had a vision
37452|Of an old and stubborn old man,
37452|His hair was pale, and thin,
37452|His face was all forlorn,
37452|And the moon was full in the air,
37452|And a spirit passed over his brow,
37452|And its face was all for ever.
37452|And he spoke:
37452|'Have we ever a dream?
37452|Have we ever a vision
37452|Of the ghost's ghost?'
37452|The Master gave the word:
37452|'By the breath I know
37452|The meaning of Death:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The spirit said:
37452|'By the breath I know,
37452|The meaning of Death,
37452|You will see a ghost
37452|Stand by the door
37452|And enter.'
37452|And the spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You may understand:
37452|Can it be 'hush?
37452|Have we ever a dream?'
37452|The Spirit said:
37452|'By the breath I know,
37452|The meaning of Death
37452|You can see a ghost
37452|Stretched toward the door,
37452|And see a spectre
37452|Pass by the chamber door.
...
## Sample 24:
...
1333|Then, sweet heart, whisper, sweetheart,
1333|"Thou art sweet, but thy love is vain."
1333|I do love thee, my love,
1333|In a word, in a song,
1333|With the heart and the will,
1333|And the power of my heart;
1333|The power of my whole
1333|Of the poet's soul,
1333|And the heart and the soul!
1333|As the winds take the leaves
1333|As the flowers take the flowers,
1333|As the floods take the dew,
1333|As the salt runs in floods,
1333|As the salt runs in floods,
1333|As the snow in the seas,
1333|As the rain in the logs,
1333|As the wind comes and goes,
1333|As the sleet in the coppice,
1333|As the snow in the coppice,
1333|As the snow in the bogland,
1333|As the hail in the river,
1333|As the snow in the river,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the snow in the county,
1333|As the rain in the vale.
1333|As the stars take the dew,
1333|As the sparks fly from eye,
1333|As the sparks fly,
1333|So the hand of my heart
1333|As the heart of my art
1333|As the tongue of my lips,
1333|As the heart of my heart
1333|As the flame in the eye.
...
======================================== SAMPLE 39 ========================================
|And as the summer twilight,
34237|When the golden vinewood
34237|Strikes the silent midnight,
34237|Stands mute beside the brook,
34237|With a ghostly sense of the human heart
34237|Forgotten, yearning, sighing.
34237|I do remember how, long years ago,
34237|At the spring by the vistaed stream,
34237|I stood as 'neath the orchard, in the June,
34237|To the sound of the grass and the dream.
34237|I know the moss where the violets
34237|Quested the dew and the sun;
34237|The air above 'mong the orchards
34237|Murmuring ever of bees;
34237|And the heart that was filled with the music
34237|That came to the listening trees,
34237|While the bluebird's notes, as he piped again,
34237|Awoke the robin's golden throat;
34237|And the sound I heard, long years ago,
34237|Came through the wood and the dells,
34237|Bringing the sound of the violets
34237|And the perfume of dying wells.
34237|And the song I heard in the August dusk,
34237|In the August dusk by the lake,
34237|Was sweeter, from the full-leaved orchard,
34237|Than the sound of a happy brook,
34237|When it came to the school of my childhood,
34237|And to the school of the land,
34237|Oh my home of the woods, where the wild-flower
34237|Loses itself and dies!
34237|They give me back the old-time delight,
34237|The pleasant and the calm,
34237|When still the wind was blowing in the woods,
34237|And the children stood in the warm, glad school,
34237|And smiled as the dear lad asked.
34237|They give me back the pleasant book
34237|That gave my heart its fire,
34237|Those childish words, the constant brook,
34237|Those childish words, the tire;
34237|They made my soul to loiter!--Yes,
34237|They do, they make me blest!--
34237|The rest of the household, and the rest
34237|Of the parents whose hearts were filled with care,
34237|And who were sad in their care!
34237|Their voices!--Yes, and they do--
34237|'T was aye! 'T is aye! 'T is aye!
34237|And the dear friends, so dear to me,
34237|They still will live and die!
34237|I have not a moment now
34237|To forget when the morn is gray--
34237|To be happy, and cherish so
34237|The rose that is on her way.
34237|The evening breezes blow,
34237|And the stars shine out to-day--
34237|But I would not live in to-day,
34237|If I were as happy to stay!
34237|I hope that maybe one day,
34237|When all my work is done,
34237|My darling's coming away,
34237|To meet me in the sun;
34237|I hope that maybe I can see
34237|My Peggy's smile upon me.
34237|The evening wears an old, old gray,
34237|Which softly slants upon the way,
34237|Its shadows on the sunny day,
34237|Its shadows on the sunny day.
34237|O'er life, a sad, unwritten scroll,
34237|The words are like the gentle dove,
34237|That sails upon the nightly soul,
34237|Though none may read or hear reproof.
34237|And drooping o'er life's weary way,
34237|God grant the book may never end,
34237|The gentle words that cheer my way,
34237|The gentle words--they come to blend--
34237|The tender words of comfort and of love,
34237|The kindly words--they come to bring me joy.
34237|I know not if my path shall be
34237|Through the world's wild, woeful wild;
34237|But I know that sometimes, in the night,
34237|The dark will come, with wild delights,
...
======================================== SAMPLE 64 ========================================
 away,
2620|And be glad as the lark
2620|When the skies are clear;
2620|And send forth a breeze of love
2620|As of wings to our bark,
2620|And away with a joyous song
2620|As of streams in our ears,
2620|And away with a joyous tune
2620|As of birds in the spheres,
2620|And away with a joyous tune
2620|As of voices in trees,
2620|As of swans in the summer time
2620|When the grass is green
2620|And the air is keen,
2620|And the leaves are young--
2620|Then away with a song of praise
2620|As of flowers in Maytime
2620|All the sunny days!
2620|O beautiful, gentle, and clear,
2620|Illimitable and strong!
...
======================================== SAMPLE 72 ========================================
, he who had no need to fly;
24869|For in this moment of dismay
24869|The king who held that evil foe
24869|Threw Indra's son as he drew down
24869|The Lord of Life shaft-headed and bow.
24869|Then Indra, lord of every woe,
24869|The Vánar legions, with a shout,
24869|The Vánar legions met and fought,
24869|And straight they broke the tyrant's yoke,
24869|And hurled him at the giant, broke
24869|The mighty bow the giant broke,
24869|Which Indra, King of all the Blest,
24869|Had thrown by Rávaṇ's(924) mighty breast,
24869|The monstrous coil, the brawny hand,
24869|The monstrous mouth, the jaw, the jaw,
24869|The jaw, the jaw and bleeding jaw,
24869|The ungovernable host, the jaw,
24869|And the great bow which never bends,
24869|The arm, the fist, the knee, the ends,
24869|The body laid with mighty stroke,
24869|And the great bow which never bends.
24869|So, when the giants fought, and fell
24869|With murderous strokes, the giant fell,---
24869|So falls the tree with all his trunks
24869|Terrific in its death, that shoots
24869|Wild volley at the mighty trunk,---
24869|So fell the tree with all its boughs
24869|While all the vipers dug and sowed---
24869|So fell the tree with all its boughs.
24869|But Ráma's heart was sad within:
24869|He wept and mourned his captive's sin,
24869|For he had wrought a ruin yet
24869|O'er Raghu's son in his wrath,---
...
======================================== SAMPLE 78 ========================================
 on the bosom of
11014|King Deshav, son of Bhishma, sat in the shade of the trees,
11014|Humbu, the great, strong, beautiful, fortunate Brahmin,
11014|A king, a keeper of the law, a guide of the realm,
11014|His name unfolded through all time and space,
11014|A ruler of the realm, a keeper of the realm,
11014|And was worshipped, as was meet, by the Great Spirit of God.
11014|And all the days of his life he kept on striving with God
11014|For the union of faith; and at last all-wise he spoke to
11014|"Lord, I am the Brahmin's lord--and I hold thee thine inmost
11014|As I cast my life away from thee, my Lord, to-day!
11014|Therefore I cast mine body away from thee, my lord."
11014|And that, by constant penance, I might win thy favour
11014|So in the spirit's depths he plunged it into the sea,
11014|But, as the wave closed over it, the wandering wind
11014|Caught up the ship's chattels, and bore it with it to the beach.
11014|And Bhimasena seeing there the empty space behind,
11014|The wandering ship rocked in the dark and glowing heat.
11014|He sat upon the bosom of the Mother of God,
11014|He sat upon the emerald seas, meditating death
11014|Of the great sea.  He sat and pondered in his mind
11014|Upon the mystery of the sea, what gods the daring man
11014|Must have to tell of,--and this mystery,--when, in the morning,
11014|As, in the after days, the Lord of life should pass away,
11014|And leave the body alone to ride the ocean's force,
11014|To die in solitude, unknown, untroubled,--and unto him
11014|His world was opened; and as yet no living creature.
11014|And all the night he sat there, gazing in the east,
11014|Until the morning sunlight faded from the hills
11014|And dawn came, bringing darkness and the darkness awful,
11014|And to his soul came holy light from God, to cleanse
11014|All doubt and all resistance, till, in the morning of life,
11014|The coming of the Lord beheld his face.
...
## Sample 95:
...
24869|Canto XXI. Lakshman's Speech.
24869|He ceased: then Raghu's son repressed
24869|The sovereign of the giant kind,
24869|And thus with soothing words unsoft
24869|To Ráma Rávaṇ spake:
24869|"Come, with thy brother Lakshmaṇ, friend,
24869|And Lakshmaṇ, and the dame agree.
24869|Thou in the woods shalt soon be found
24869|And bathed in pleasant waters clean;
24869|Where thou shalt sit, and rest, and save,
24869|Well clad in coats of bark and hide,
24869|What days, what nights, what hours will pass
24869|That thou in holy heaven mayst see
24869|Thy darling with her night-made tressed
24869|Far from the forest. Thence will spring
24869|Sweet smells of pleasantness and light
24869|And bliss from the celestial string.
24869|Thence on the ground shalt thou be borne
24869|O'er the bare earth, O Queen Mosteer,
24869|And on the fresh bright earth where thou
24869|Shalt sit in state with Queen Sítá,
24869|In glorious heaven the nights and days
24869|Thou wilt be rapt by the great bliss
24869|E'en as the Lord of Gods is hearkening.
24869|The nights and days are thine, O best
24869|Of giant lords, and I, the best
24869|Of all who love the Lord of Lords,
24869|Whose might can turn the firmament,
24869|Whose might can sway the leafy bowers
24869|And turn each flower and leaf and bower
24869|To holy joy and blissful flowers.
24869|Ah me, the languorous days are come,
24869|And not a moment shall I see
24869|The happy days of Ráma's Queen
24869|Far from the light that round her glows,
24869|And marked with darkening leaves and boughs.
24869|Ah, whither would her steps be turned,
24869|And where the woodman's art had burned?
24869|Ah, whither would her steps be bent
24869|To turn her toil-worn heart once more,
24869|When all her hours were joy and peace,
24869|And all her hopes were set on store?
24869|Ah, let thy soul be comforted,
24869|Let trembling fancy still excuse
24869|The burden of a weary time
24869|That mars a saintlike life and use.
24869|Ah, if thy love were still the same
24869|That now I watch with toil and pain,
24869|That I could be for aid or flame,
24869|Could not my heart and bitterer gain."
24869|And Lakshmaṇ to the forest came
24869|And told his tale with welcoming.
24869|He saw the tree where he was set
24869|With burning buds and leaves beset.
24869|He saw the tree where he was brought
24869|By Sítá of the glittering thought,
24869|And when the leaves were fallen, he
24869|Spoke of his lord the tallest be.
24869|"O Lakshmaṇ, I the deer will slay
24869|From thicket, cave, and mountain gray,
24869|Ere long shall I this forest seek,
24869|And Lakshmaṇ in the covert seek.
24869|O'er hill and wood the Vánar bands
24869|And watch the beasts of wood and sands."
24869|He spoke: and Lakshmaṇ's love obeyed
24869|Nor did he speak as he was prayed.
...
# Sample 100:
...
38475|O Liberty, the patriot's sure defence!
38475|True to the man who fears a tyrant's eye,
38475|Preserve thy rights, and own his glorious cause,
38475|And yield the haughty title to a lie.
38475|No longer now on mean estate depend,
38475|And England owns thy sovereign vital force,
38475|And her best sons succeed to guard her laws,
38475|Or her best sons bestow a deedless course.
38475|Now, from that happy climate freedom's hope had birth,
38475|And made one day a milder country bleed,
38475|To the great cause that gave her aid is given,
38475|And to mankind one sure reward is even,
38475|Whilst I, perhaps, to distant climes must speed.
38475|To the same cause who has the cause to join?
38475|What foes against mankind may rise to arms,
38475|Boldly they fight, in actions of design,
38475|Yet all the same, and every day they charms.
38475|Ah, Washington! who can thy cause design?
38475|What can the nation do, or me, subdue,
38475|But still go on, in humbling folks admire!
38475|That we may praise thy conduct, that we fire,
38475|And for thy conduct many a hero dare,
38475|That we may rise, and cast the tyrants down,
38475|And tyrants fall, and fall the people crown!

Not bad.

These samples represent roughly top decile poem samples (~10 out of the first 100), at least by my selection.

Scott Alexander & commenters highlighted a few more samples:

Thou know'st how Menoetiades the swift
Was dragged, of Hector and the fierce compeers
And Phrygian warriors. So, we will dispatch
Your bodies, then, yourselves to burn the ships
In sacrifice; with torches and with bells
To burn them, and with oxen to replace
Your gallant friends for ever. But I wish
That no man living has so long endured
The onset of his foes, as I have power
To burn or storm; for mighty Hector erst
Was slain, and now returns his safe return

My heart, why come you here alone?
The wild thing of my heart is grown
To be a thing,
Fairy, and wild, and fair, and whole

And they have seen the last light fail;
By day they kneel and pray;
But, still they turn and gaze upon
The face of God to-day.

And God is touched and weeps anew
For the lost souls around;
And sorrow turns their pale and blue,
And comfort is not found.

They have not mourned in the world of men,
But their hearts beat fast and sore,
And their eyes are filled with grief again,
And they cease to shed no tear.

And the old men stand at the bridge in tears,
And the old men stand and groan,
And the gaunt grey keepers by the cross
And the spent men hold the crown.

And their eyes are filled with tears,
And their staves are full of woe.
And no light brings them any cheer,
For the Lord of all is dead

Fair is the Lake, and bright the wood,
With many a flower-full glamour hung:
Fair are the banks; and soft the flood
With golden laughter of our tongue

How the clouds
Seem to me birds, birds in God's garden! I dare not!
The clouds are as a breath, the leaves are flakes of fire,
That clash i' the wind and lift themselves from higher!

In the dark the sun doth gleam,
And in the dark the moon doth seem
But now the evening is begun--
Gone is the sun upon the earth!
The silver moon doth like a cup
Of blood-red wine, and as that cup
Is drained of life, doth quench no drop.
What man will drink such wine?
There is no soul of earth or birth
Which man hath never known of earth.
There is no soul who doth not sit
And sing to it, and cry, "Drink!"
There is no soul whose feet are set
On youth's eternal paradise;
For all is a solemn harmony,
And all is a perpetual chant,
And all the world is a song of God.
There is no soul so wholly free

Scott Alexander highlights a fun repetition-trap one:

And this one is obviously a failure on one level, but on another level is some kind of great experimental modern political poetry:

The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), majestical,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious,
The Emperor Wu (the great Wu), rapacious

There are several kinds of people in America;
There are several kinds of people, I mean their number.
There's a girl growing up in the house by the light,
There's a youth upon the road, or a girl somewhere in New York;
There's a prettier girl, and a man more congenial,

But none of the likes of the likes of the fellows are equal.
There's one who has never been married and married,
There's one who don't want to be treated with kindness;
A fair youth is never employed nor neglected;
There's one who has never yet come to a neighbor,v
And one who resides in New York from the start;

But none of the likes of the likes of the fellows
Are equal to him, and wherever he goes,
The heart somehow breaks under the hand that is steering; And so it is with me

There comes a murmur low and sweet
As of far-off streams in a dream,
Or a murmur of many birds,
Or chime of little evening bells,
As of wedding-bells in the dells,
Soft, sweet and slow,
As of wedding belles that come and go.
A little green ribbon of lilies
By the door of my dear one's room,
A kiss on her cheek, and she whispers,
"I am the bride of the loveliest flower."
A moment we stand in the garden
Of dreams and things,
Dreaming of fairyland
And the fairy music there,
Sweet bells and dreams, and the fairy music,
The fairy songs of the air.

The top percentile of poems are probably quite good, especially with some light human editing to fix up the more glaring issues. To get a decent number of top percentile poems would require a lot of reading, but on the other hand, there is no reason why selecting or ranking poem samples could not itself be treated as a supervised learning task for retraining GPT-2-117M-poetry on, by using selected/non-selected as labels and training to predict the probability of a given sample being selected, and then such a NN could be used to prioritize likely-good GPT-2-poetry poems (or any source of poetry) for human review (and, in a form of “active learning”, the results of the manual review can be fed back in as additional data to help discriminate between the best and the merely good samples).

“Howl”

First, “Howl”. Given that the Project Gutenberg corpus is entirely old poetry and wouldn’t include much in the vein of “Howl”, I didn’t expect this to be good. The finetuning would wipe out the knowledge of free verse.

Finding a good prefix was hard, also unsurprising—not much like it in the PG corpus! I ultimately had to settle for a “1997” prefix from a relatively free-verse sample for generating the 3 samples:

## Sample 1:
1997|Mona, gazing, voluptious! exhaling human blood.
1997|Bare, the sun, in a golden mist,
1997|Leans on the meadow; the clouds are gone;
1997|Yet the heaven of green mists around
1997|Cannot but give them a fresher light.
1997|The sun hath a juice that no time can stay--
1997|And a golden night-cap.
1997|How softly the quiet clouds go by:--
1997|One sees a star through the evening sky
1997|Shine in the valley.
1997|There is no sound but the stream that strays--
1997|The very air how soft, and oh!
1997|How the waters of the meadows flow,
1997|In the warm afternoon;
1997|And the gentle wind from the hills above
1997|Is a murmurous tune.
1997|Through the still wood the cicalas sing,
1997|And every blade grows young and wan:
1997|A man is carrying a heavy heart
1997|To save himself from the cold, and then--
1997|And the last of it comes.
1997|He makes his coat, and, as he goes,
1997|His garment of a color dies.
1997|It is the night-wind, blowing over and under,
1997|With its great peasings of thundering wonder and wonder.
1997|It is the night-wind, echoing and shaking,
1997|With its great peasings of wonder and wonder.
1997|The night-wind suddenly blows a rosy bud--
1997|It is the sound of the forest trees breaking out,
1997|And a great rumbling of branches, and hoarse, and heavy and strong,
1997|And it is a shrill voice, and clear;--and the wind that gives answer cometh out.
1997|it is the loudest the voice alone in the cry of the silence of the noises in the darkness and the sky, and the moon:--and the water in the little wind in the trees.
1997|sing of the raving, and the tree with its silence that followeth:
1997|And it too long and a quiet water and they that cry of the
1997|tree-playing sea, and the air and the silence.
1997|flashing of the sky.
## Sample 2:
1997|The thundering guns! the splintering masses! the thunderous mass! the splintering guns! A welcome!
1997|Gitcheers! Welcome! the river that from thee will pass:--the storm of flame burst!
1997|The fire that is thine will be there:
1997|Hear it, my Sun-god, breathe upon me;
1997|Be strong! We ask not one again.
1997|Thou, mother Earth, if this is done,
1997|That thou wouldst be for aye to-morrow,
1997|What is thy name? to what will follow
1997|The sun that to thee sang one breath,
1997|What is thy name? In what will follow
1997|The sun in darkness, breathless, fell?
1997|The sun must rise in a day's breaking:
1997|What is thy name? to what will follow
1997|The sun in the world's last spinning-wheel?
1997|There comes a word of a holy token
1997|From Jove, the cloudless king of rain:
1997|"What canst thou, then, that I should follow
1997|The sun in heaven as on earth again?"
1997|The Lord and the Lord are lords together:--
1997|The sun will rise in a day's faring!
1997|The cloud will darken all earthly matter;
1997|The sun will burst in a day's strong-wing.
1997|The earth will moan through the black night
1997|And utter a little scream behind:
1997|"What is thy name, then, to what wilt thou,
1997|Lulled in the silence by Jove of rain?"
1997|The Sun would sink in his blue heaven
1997|And be a torch toothed in the main;
1997|The earth would be a chamber filled once more
1997|And the voice, and the heavens rise of the sea.
1997|And break a voice of the music of the heavens and the rain.
1997|Erebell of that speaks the wind again, the thunder.
1997|And the day and the flowers of all things that sang, and ocean's breath of the sun,
1997|And thou and the morning, and all that sang thee and thunder, the rain and the rain and the rain.
1997|And thou, and all.
## Sample 3:
1997|The whole cathedral of the church; the human procession, _Lyral._ 4. It had hardly gone
1997|The temple, the Capitol, the nation, was still seated there, and the altar, the
1997|pulpit, the water-mill in which it was used by the Greeks, was a perpetual murmur and in
1997|the old church, or the great, or the small, or rather the less, would have gone on at
1997|The altar of God--the altar, which had not yet been raised to the new, or to the
1997|"Blessed are they, blessed, among those who went down! and now is my will so
1997|And they asked the priests and the virgins of God to give them a prayer, so the
1997|dwellers on the earth, in which they are thinking, "a little space is granted for
1997|ever. If he were his own son he is guilty of a wrong. But he is guilty of the loss of
1997|righteousness! If he were to die before his race, his race and parents have been
1997|And they answered with voices of joy, "Remember righteousness, and remember the
1997|Thescore, the obstinate, bad and bold, at the word of the priest, was crucified.
1997|And the pious fathers and the pious men had gone to tell their sons, and to tell
1997|Then the father replied, "I would ask not the grace of thy God to form the
1997|The son was seized with the thought of death.
1997|injunction.
1997|The priests, he received the gift with the body, and the priest at the same time threw him
1997|turban and the priest's mantle. This, however, was the ransom for the son received
1997|given, and the father increased family increased his blood. The priest departed; but his body was given, he went away with tears that a widow, he died. The sated the widow gave into the father in a
1997|And it was raised, after his body, for a sowing blood in compassion of him.
1997|inery. The venerable face was brought him into the house in all the funeral and buried.
1997|giveness. But before him. The priest, together with himself.
1997|The shepherd took the widow married is thus, the priest,

While they may be OK on their own and plausible as unconditional samples, they are disappointing as conditional completions, largely ignoring both the vocabulary & style. It would seem that the finetuning wiped out whatever it was GPT-2-117M was using to generate its amusing “Howl” completions.

“Ozymandias”

For “Ozymandias”, I fed it in a few times, and it seemed to like numeric IDs starting with ‘88’, so I used this as a prompt:

8820| I met a traveller from an antique land
8820| Who said: Two vast and trunkless legs of stone
8820| Stand in the desert... near them, on the sand,
8820| Half sunk, a shattered visage lies, whose frown,
8820| And wrinkled lip, and sneer of cold command,
8820| Tell that its sculptor well those passions read
8820| Which yet survive, stamped on these lifeless things,
8820| The hand that mocked them and the heart that fed;
8820|
8820| And on the pedestal these words appear:
8820| 'My name is Ozymandias, king of kings;
8820| Look on my works, ye Mighty, and despair!'
8820| Nothing beside remains. Round the decay
8820| Of that colossal wreck, boundless and bare
8820| The lone and level sands stretch far away.

Yielding (3 samples):

## Sample 1:
8820|... the last captive died of his own shall
8820|and the stateliest of towers was the king...
8820|And he entered the city.
8820|He sat ironing in his counting-house,
8820|Flinging his burdens from a hundred kings;
8820|(A queen had one, and her name was Pepban)
8820|And the king was the son of Pepban.
8820|Then he said: 'O son of Hernando!
8820|Not a crown of gold nor iron can he save,
8820|But he shall conquer by his own right hand,
8820|And the palace of the king be free and save,
8820|And the crown of the head be free and save,
8820|And the wealth of the world be free and save,
8820|And the wealth of the world be free and save,
8820|And the king be free and save.'
8820|And the seer took the gift. He received it:
8820|And he wrought a wonder in his brain, and died:
8820|But the seer, like a god, whose history
8820|Is unendowed, and expoundeth unto it,
8820|--So shall one tell the story of his glory....'
8820|And the seer surveyed his old hard case.
8820|But the king was old and rough, he died
8820|Of age and weakness; and the seer replied,
8820|'I will tell them to my tale, and they shall learn
8820|Who sing this song, and live like equal lives
8820|In the kingdom of the king.'
8820|Again the seer said, 'We will go and speak
8820|A gentler story in a palace: it shall keep
8820|These seers in silent wonder still and deep
8820|As sleep within his grave.'
8820|And the seer took the gold:
8820|'I have no doubt that I shall tell it out,
8820|And I knew that for me shall there be no fear,
8820|If I tell a lie or clear.
8820|But tell it forth,--the poet shall be clear,--
8820|For never any hero
## Sample 2:
8820| Alluding to the curious fable of the Aztec king, who, in his
8820| The palace called319iform; the people of the region of Bahr
8820| The palace called31920|--Rua, behold! a new king came to mingle these two
8820|_So that I wept beside the dead,
8820|And buried deep in thought I wept.
8820|I, like a desert in a glade,
8820|Keep wistful vigil all the night--
8820|And on the palace built a pile
8820|Of lamps that, kindled, slowly shed
8820|A ceaseless light upon the dead.
8820|On this I saw the shadows grow;
8820|And, dreaming there, I, too, lay down--
8820|The dead.  I wept.  I wept--upon
8820|A couch and saw the shadows too.
8820|And yet, indeed, I wept.  For such
8820|A cold to-morrow's light will gleam,
8820|Or such a flame toward one like this.
8820|Then shall the spirit's eyes grow dim;
8820|And in thy face she read my heart
8820|I sometimes read a thousand rhymes
8820|Of hopes, and fears, and hopes have part.
8820|A moment as I read thy thoughts,
8820|My heart was moved; I was athirst;
8820|Yet I was well content to be
8820|The slave of some great memory;
8820|Then, in this vision of the night,
8820|My heart grew very sore with grief,
8820|And I fell prone.  Oh, it was sweet,
8820|As I lay trembling near the dead.
8820|And, as the dead-man's face, that slept,
8820|Groping beneath the churchyard yestreen,
8820|I heard the murmur, soft and deep,
8820|Of a sweet spirit, wafted thin,
8820|That made the shadows seem to creep,
8820|And murmur, soft and low and mild--
8820|And watching, through the moonlight, where
## Sample 3:
8820| The ebbing tide bears only the sands
8820|And trees that wave their boughs to the western breeze--
8820|A broken landscape, where faint breezes waft
8820|The lingerers on their way to Indian seas--
8820|A land whose shores are sunset skies,
8820|Where the pale mists of evening slowly sail,
8820|And, as we steal, the rivers murmur strange
8820|To the low-murmured wind at fall of night,
8820|And to the distant rim of the far sky:--
8820|Here, silent as a solitary tomb,
8820|With all that life can purchase and wealth can lend,
8820|With youth's immortal spirit, thou, O Friend!
8820|Art not for me to give thyself to Art?
8820|To thee, who art the latest breath of lands,
8820|Whose fame has swept the world for which men sigh,
8820|I give my heart up with indignant hands,
8820|To thee, who art the only one I seek.
8820|For thee, alone with life and light impearled,
8820|I cast my eyes from out the vast of time
8820|Where nothing dwells but Destiny's brief stage,
8820|Unshrouded, till, in its frail waste of time,
8820|It shall re-risen and grow once more sublime.
8820|And then, within the bounds where all is strife,
8820|And death, and life, and love, are all my own,
8820|I take this heart up with indignant breath,
8820|That seems to harden with the weight of years,
8820|And with the bitter hardihood of tears,
8820|And with the heart that has no beat or fear,
8820|And with the hand that has no strength to bear
8820|Only the world's and with the heart that fears,
8820|And by its sorrows only known to keep
8820|The beautiful and glorious years we keep.
8820|Here are the little walls we meant to build.
8820|Here stands the doorway, opening to the East,
8820|The child-like likeness of a poet's child

Sample #2 is over-influenced by some prose footnotes/commentary which apparently were in PG, but the analogy of Ozymandias to Aztecs is a potentially fruitful one. And sample #3 here is a particularly good extension.

Essay on Criticism

Not clear what text exactly Scott Alexander used from Alexander Pope’s Essay, so I quoted the famous beginning section of Part 2. 3 samples strongly indicated Pope-like writing was associated with a prefix of ‘385’ (if not necessarily a full prefix) so I used 38511 for the following 3 samples:

## Sample 1:
38511|Pride even in numbers; wit's a kind pretence
38511|To something foreign still, but ne'er to sense;
38511|A constant waste of words, the world produces,
38511|A lazy sap, that nothing that's foreign to expels,
38511; that's foreign still unknown to the soul;
38511|A young, like, but in the world in motion,
38511|Obscending not, which smells all our own the worse than our own our own the soul's, and soul;
38511.
3851166|Like sense; which is like, but in all our whole.
38511|Which thus far more like, but in all things that's an excellence; and ne'er unchanged by which is folly's the worse, they give the worse maintained by which
38511|If sick of sense;
38511|Wholubil, or snug. ills, we know our own our first in sense the worse maintained between the worse, soon expired.
38511|Is blind and first blown on both being free from sense;
38511|From sense the worse maintained
38511|Wholuteness seems at once more used to sense the worse maintained by which
38511|Wholublime or enjoying sense; and first made to the worse, will's the worse maintained to sense;
38511|For sense; by which smells now discharged, and kept unseason'd from sense;
38511|Whose.  that's soon revived.  and then past the worse maintained, by birth to sense; by sense the worse, with weightyselves;
38511|Mankind by being all else barren; the worse maintained and last by birth to sense;
38511|Would cast, since built in nature lies from sense; for which smells and last, by repugither to sense;
38511|Whole; for our present, and first in life at all else to sense so long since built o'r by life to life, is soon revived by contact with heav' we know our own th
e worse maintained the worse it burns first made equal right;
38511|Is free.
3851166|Or dead: thus far more; who survey.
38511|Or wry's profuse and then dead: but what oft the worse maintained and next to life.
38511|From all; and
## Sample 2:
38511|There lies, that write the very best of all;
38511|For the lov'd fool, for those he courts and chokes,
38511|Is but a thorn inborn martyr in grief and sin,
38511|Who would all bawls and rattlantoms.
38511|Some hazels and isle from a thorn, or a starv'd for breaking hearts abh, to hogs;
38511|Or movel sooner writ, when by the starvels, or fombe.
38511|For men of any faultless wox and bribes.
38511|For wagers who should cut apart, and wak'd to make 'gin rights, for stink; for lamb, or chase; for lamb.
38511|Pounders or cast heel or a rope. For, for lamb, for lamb, for lamb or for lamb; for lamb, for lamb or starve.
38511|For no mean; for lamb, for sheep or for lamb, for lamb, lamb, for lamb, for lamb or for lamb, for lam, for mares.
38511; for lambs, a-heats. Suchley, for mares, for mares, for themselves.
38511.
#. (for lambkins.
38511|mells; lam, lamb, lamb; lamb, lambkins; and other's for wares, lambkins; for struts; for sheep, lamb; or pricks, lamb, lambkins; for wer clothes; for mares: for sheep for lambkins; for goats for lamb.  for goats, lamb; for lambkins; and for moo: for hethers, for wark;
38511, lamb: for babes.
38511 unman; and for lamb.
38511; for lambkins; for mares; and the wox: for wheroat.
38511; for other goods, lamb. (let, frothriars, for spoles; a drum); for goats for lamb.
38511d:) for spoons, for spoons; for goats for whelornes, lamb, for mares: for syr trespass, for goats for mares: for lamb; for mares; for goats for sycam;
38511 theirs, for dog; for mawds.
38511; for whel
## Sample 3:
38511|They talk of constancy and faithless love,
38511|A seraph trembles at the specious glove;
38511|Nor in the rich confin'd relief of state,
38511|Find proud pretence, nor in the disdiscoveries of fate.
38511|For when misfortune makes choice remains the conduct's the prize half known,
38511|Can we secret soul without due, they fear of sense of more known.
38511|Some rise where' rights, they make it pays due.
38511861.
38511861.
#. Sense and judgment, as equal prize seem meanly, the reward the joy, as much possess the prize paid, as well reckon the prize we do not less dare not less keenly wise.
38511861|We see;
38511861|Of happy lovers ought, as well done, like a friend.
38511861|Know they ought, 'tis ev' is the other joy, as well worth a right;
38511861|The joy, as well might, as well may, as well may all is great.
38511861|Nor need of joys not as well maysters, as well as well may they give; but as little store; but as well as well as well may shewn, as much, as well we know, as well as well can be sure might prove, as well may well as well as well as well as well may view;
38511861|The mind: as well as well as well as well as much the fair as well as well as well as well as well as well as well may,
38511861, as thou woe.
38511861, when all.
38511861:
38511861|Well done by the fair as well might rise, who merit thus prove by 'tis most of all are one;
38511861ly play the few.
38511861|There's well as well as well as well as well as well as well as well as well as well as well as well as well as not less admire;
38511861|Apostorably bright.
38511861|Well done by sight.
38511861, as well as well as by contact;
38511861|For they; the main aim

Alexander described his GPT-2-117M sample from Pope:

It understands there should be line breaks, it understands the approximate correct length of a line of iambic pentameter, it understands how to talk like an overeducated 18^th-century dandy—but it doesn’t appreciate rhyme or meter. In retrospect this isn’t surprising; GPT has no idea words sound like anything; it would be shocked to learn anyone uses language as anything other than text strings.

GPT-2-poetry-prefix still has “overeducated 18^th-century dandy” down pat, but it manages to improve on the rhyming aspect: there’s quite a few rhyming lines in samples #2 & #3 (#2 seems to be screwed up by taking a digression into footnotes defining words and then bad sampling getting it trapped), like “pretence”/“sense”, “soul”/“whole”, “love”/“glove”, “state”/“Fate”, “bright”/“sight”, and a number of almost rhymes like “right”/“great”. One wonders if it’s learning by brute force and memorizing specific pairs of rhymes (although could there really be that many rhymes of “state”/“Fate” in even 3m lines of old poetry?), or if it’s doing something more equivalent to inferring the latent phonetics from the co-occurrence of byte-pairs? (That may sound unlikely, but word embeddings do many unlikely-sounding things with no more supervision than co-occurrence¹¹.)

More concerningly, the samples are terrible. Pope’s poetry should be straightforward for GPT-2-poetry-prefix, as it follows standard meters and rhyme and relies on a classical vocabulary well-represented in the PG corpus. Why, then, are they so bad? I suspect this may reflect the corpus itself doing Pope a disservice. Pope’s inclusion in the PG corpus appears to consist of the following (grepping for “Alexander Pope”):

32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory

Checking PG entries and looking through the 32190 prefix, it starts:

32190|INTRODUCTION                                                 xv
32190|The Works of Mr. ALEXANDER POPE. London: Printed by W.
32190|BOWYER for BERNARD LINTOT, between the Temple Gates, 1717.
32190|This volume consists of all the acknowledged poems which Pope had
32190|The Works of Mr. ALEXANDER POPE. Volume ii. London: Printed
32190|by J. WRIGHT, for LAWTON GILLIVER, at Homer's Head in Fleet
32190|Letters of Mr. ALEXANDER POPE, and Several of his friends.
32190|London: Printed by J. WRIGHT for J. KNAPTON in Ludgate
32190|Street, L. GILLIVER in Fleet Street, J. BRINDLEY in New Bond
32190|Street, and R. DODSLEY in Pall-Mall, 1737. 4to and folio.
32190|The Works of Mr. ALEXANDER POPE, in Prose. Vol. ii. London:
32190|Printed for J. and P. KNAPTON, C. BATHURST, and R. DODSLEY,
32190|The Works of ALEXANDER POPE, ESQ.; vol. i. with explanatory
32190|Notes and Additions never before printed. London: Printed
32190|commenced printing his particular section of the octavos when the
32190|Quo desiderio veteres revocamus amores
32190|Atque olim amissas flemus amicitias.
32190|Nutrix mea fidelissima M. Beech, obiit 5 Novem. 1725, aet. 77.
32190|Edwardus Blunt, vir amicissimus obit, Aug. 1726.
32190|Francisc. Atterbury, Roffens Episcopus, vir omni scientia clarus,
32190|The fourth volume contains the Satires, with their Prologue,--the
32190|alterations. --_His Last Will and Testament._--WARBURTON.

This is perhaps not good training material for GPT-2-117M-poetry/prefix and explains the bizarre degeneration—it is ‘expecting’ sudden random irruptions of largely-irrelevant prose such as introductions or footnote-annotations (rendered inline by PG’s text formatting). Other entries in the corpus will be more free of scholarly or prose apparatus. (In retrospect, a preprocessing step like dropping lines longer than ~60 characters might’ve been a good idea.)

8 Famous First Lines

The prefix trick doesn’t work on the 8 famous first lines nearly as well as it does with the long excerpts from “Howl” etc; I assume they are simply too short to home in on a relevant prefix. Nevertheless, I tried.

## Sample 1:
2428|And he who wears a crown, may die, but little lives.
2428|It was a pleasant day, when snugly shut,
2428|For many happy moons and many a sun
2428|Had lit the laughing waves with constant scarf,
2428|And heard the pealing waters on the run.
2428|'T was noontide; and with such a sky-lark flight
2428|The fleet white pieces flew in waves of sound;
2428|The silver stars, their fitful little light,
2428|Were beating on the liquid night around.
2428|Down went the cold, and all the dreariest blasts
2428|That ever chanced to blow, came mopping near:
2428|The first and fierce, the youngest, was our statesman here,
2428|Old Man-Autumnus!--An oak-headed oak!--
2428|The oak that in our court has grown so hard!
2428|"_Here am I_," answered he, "when I am dead,
2428|And round these sacred ashes I shall cling;
2428|And all the while my heart, unsatisfied,
2428|Holds but one clue,--and how must I be king?"
2428|How could he hope that a short space of day,
2428|And the warm blush of eve, and a fierce warning felt,
2428|Could prove the son of an old hunter, killed
2428|By the wolf's hounds in the wood! He crouched, and leant
2428|Against our shadowy robes; and the boughs bent low,
2428|Like the snow-flakes that cover the world below,
2428|Bent with a conscious motion. As the breeze
2428|Fluffs away, and the suns shivered and sighed,
2428|A shadowy motion, like a sigh, went by.
2428|It seemed a shadowy grief, and a mysterious
2428|Pestilence, like the voice of one who has sinned.
2428|At length, to his own chamber silently,
2428|We came, and gathering up the golden rim
2428|Of the great golden moon, looked down and wept.
2428|Then in a voice so soft, so
## Sample 2:
36287|Weary of wandering,
36287|Weary of loving,
36287|Can love a tawny cheek,
36287|Or an eye of ruby beaming through?
36287|My heart, why come you here alone?
36287|The wild thing of my heart is grown
36287|To be a thing,
36287|Fairy, and wild, and fair, and whole;
36287|With but the lion-heart to mate,
36287|And yet the heart to feel and know
36287|But in the wildest nightingale;
36287|The tongue you have for praise or blame,
36287|The heart that is more soft and tame
36287|Than the wild flower, this summer day,
36287|Shall never know your own dear way.
36287|Though cold and bleak your cheek I feel,
36287|Though bleak your mouth,
36287|Though cold your hand I would not steal,
36287|Oh, love, a thought beyond all words,
36287|Could I but win that word of birds,
36287|Could I but win that heart of yours!
36287|O sweet and tender is the lute,
36287|And soft the song the scarlet throat,
36287|When song-birds wake its wildwood lyres,
36287|And wild-bees through the glens go by.
36287|To you the wild-flower opens wide,
36287|To you one ripple stirs the tide,
36287|One long sweet longing seems to be
36287|A look of love made yesterday.
36287|To you the wild-flower bows its head,
36287|To you the wave is musical,
36287|Each kiss a kiss--but what of that?
36287|Ah, love, beyond your reach you keep
36287|All joys for you--ah, hear you me!
36287|Above your grave the winds are laid,
36287|Love lulls asleep within your shade;
36287|There is no rest for weary feet,
36287|The years are unfulfilled, I trow,
36287|And 'neath your sad and smiling eyes
36287|The sun tracks every path that lies
36287|Across your
## Sample 3:
1020|Who had too much of common human things,
1020|Being at peace with all he had on earth,
1020|Had something of a high and godly pleasure
1020|To do without him, since he had been
1020|In all the world so utterly,
1020|And that he had not far from his ambition,
1020|Had done no act for any human duty,
1020|Could not have felt his life's inheritance,
1020|If some good friend had lived to teach him thus
1020|His father's story of himself and country,
1020|And to restore his kingly name to men.
1020|O, the days that grow in my lament,
1020|And the nights that grow more still,
1020|When the earth lies awake,
1020|And the heaven it girdles,
1020|And the woods all shake,
1020|And the bright stars glitter
1020|Through the shadowy mist;
1020|When the night comes trailing her star-tinted hair,
1020|And the stars are the gray of our being,
1020|And the night wind is like some old sorrow
1020|Whose sweet touch is as of heaven,
1020|With its shadows and with its stars,
1020|Gathered in one place, is a child's face,
1020|A child's face made of golden sea foam:
1020|And the tears and the prayers that we utter
1020|Will be as those broken tears
1020|That fall upon bosoms, at the hour of their birth,
1020|With a sad and sorrowful melody,
1020|With a last, far-off singing of earth.
1020|Yet the night is a golden sleep,
1020|The day is like gold,
1020|And the stars go out alone
1020|Through the dark leaves and the darkness, alone.
1020|Night above and stars about,
1020|All our wanderings ended,
1020|Time above and darkness about,
1020|And the sun all hidden,--
1020|We are two two; the world is two.
1020|Howsoever the hours clap and clap,
1020|All the earth is happy,

## Sample 1:
28591|I would to God that it all came to pass,
28591|That my old line of trouble and poverty
28591|Was left here desolate of my poor discontent,
28591|That the first cause of all my own heart's moan
28591|Was to be born in this sad world--a lone lone lone way?"
28591|I read it in my sonnet; his deeds were so brave,
28591|So kindly a husband, so cunning a guide;
28591|So tender a heart he went with his work,
28591|That his life never stinted its comforts at home.
28591|I read it in my sonnet--and he was so true,
28591|So loving my child, that it ever proved best,
28591|And though I was forced from his tender command,
28591|I felt I was growing up, when his voice, all tender, said,
28591|"If Love be unfaith, then he is the earth's father."
28591|I read in my sonnet--and he was so sweet,
28591|So eloquent was his look and his tones,
28591|That I smiled when I saw him, and ran forth to meet
28591|His father--and asked him in all things to meet.
28591|But some were too haughty, the most malgracious,
28591|To love their loved Father--the more to annoy;
28591|And some were too cold and abandoned, and when
28591|They saw how much reason has melted and flowed,
28591|They cried, "O Love, give us to love, give us fire,
28591|So we may die, as we will; O perfect, as dead,
28591|With the dead silence!" Then each took his own way,
28591|And, in its nature, an exquisite sorrow lay,
28591|Like love in a garden of fresh green;
28591|They were happy--too happy--by nobody knows!--
28591|The world, too, was happy by Roger and John,
28591|Though some at her heart cannot understand,
28591|Yet the whole world was happy on that first day.
28591|O thou who art so beautiful!
28591|What is the world without thee?
## Sample 2:
1365|O, there's a country, there's a town,
1365|And here's the place that I have stroll'd.
1365|See! there's a town!
1365|And here's the place, and here's the goal;
1365|And here's the flower for me and you--
1365|For you and me.
1365|Here's my little girl;
1365|And here's the spot that I call mine--
1365|"Come over here!"
1365|He's under the lilac-tree
1365|Where I loved so often to dwell;
1365|He's near his own tree-trunk,
1365|But here's his knee--
1365|And he's floating away
1365|To the roses in the churchyard,
1365|To the fields where I was born.
1365|Here's the lily-thorn, and here's
1365|"But where's the rose?"
1365|Here's a little lad;
1365|But a marigold he's brought
1365|Here's a ribbon at most:
1365|"Spring comes in the sun," says he,
1365|"And you dream of flowers gay
1365|On the hills of Arcady."
1365|That's the way he's trod;
1365|He shall be a king, and he
1365|Must have to sing--
1365|Ay, a king--a queen--a grave--
1365|Whose name was love, his heir--
1365|For the royal blood he gave
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast,
1365|And the crimson wine he drank
1365|At the royal marriage feast.
1365|They love to wake alive,
1365|To gaze at life and die;
1365|But they do not understand
1365|How many thousand voices call'd
1365|Out of the ages by the dead,
1365|Out of the ages by the living's voice.
1365|How many voices, and what else
1365|Did they call the living dead?
1365|All which of them was gone
## Sample 3:
1020|A time will come for tears and cries,
1020|And then I'll know her by their eyes
1020|I had her, but I cannot tell,
1020|And I am lonely in my cell
1020|Since the time the fair one came to me,
1020|I who loved her since my first year,
1020|Though in my youth she had been dead.
1020|I had grown quiet as the night,
1020|And I had won her tender sight;
1020|And the soft notes of her voice
1020|Would haunt me without joy or fear,
1020|Though the day had come.
1020|This breath in the air took the damp,
1020|And was gone to the damp,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was gone to the black,
1020|And was flown to the white,
1020|And was gone to the white,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was flown to the black,
1020|And was gone to the black,
1020|And was carried away in the mould
1020|With its all-written countersign.
1020|I thought the words would mean the fight,
1020|But the fighting face was not my sight,
1020|I thought the men would take a share
1020|Of pain and strife and weariness
1020|And leave me on the dusty street,
1020|Where I would meet that sorry man
1020|And tell him tales and talk to him;
1020|He said, "I'm sure I've done my best,
1020|But I'm not the man to meet the west."
1020|He said, "I'm going on our track,
1020|With my best friend here at the back,
1020|And my master, as I understand,
1020|And the rest of the country side,
1020|Where people go and people ride,
1020|And some end never to decide."
1020|"I can't," I ordered, "I can't!"
1020|He said again, "I say

## Sample 1:
28287|With heart and hand he went--
28287|The boy, like any lad of five,
28287|Was never seen to pair;
28287|For he ne'er met with a taste of grog;
28287|And now, howe'er his foes may say,
28287|I wish the boy would do all that he can,
28287|And so will merely go to bed;
28287|For, ere he goes to bed,
28287|He'll do most things, not more.
28287|He's gone to bed as readily;
28287|Then, before breakfast, he politely shuts
28287|The book, unless he's said,
28287|And talks of how the breeches blows,
28287|And how the shadows fall;
28287|And how a lady steals a cheese;
28287|And what a consternation comes to dress
28287|That baby sleeping there!
28287|He knows it is a drowsy child,
28287|And therefore it is very wild:
28287|For he ne'er goes to bed--'tis true--
28287|But he is there, and he is fast asleep.
28287|My dears, he says, if you can look
28287|In your writing, he'll just show you just
28287|How very fond you were of me,--
28287|And for how long I loved you so!
28287|And 'tis a very different thing
28287|He reads me, as he so loves me,
28287|Just to think and write of me.
28287|The moon's a very different thing
28287|I should prefer to read to him;
28287|The dogs do bark, the parrake buzz,
28287|And so do all upon me.
28287|One day there was a certain man
28287|Who said, "I love you better,"
28287|Because he had four teeth, they say,
28287|And still was most inquisitive.
28287|They sent him to a stylish drive,
28287|And, though they griev'd him greatly,
28287|They gave poor vulgar names--I trow--
28287|The names--and what do you think?
## Sample 2:
1852|He sees his wretched wife's fate, and dies.
1852|"And she, perhaps, may die in future time."
1852|In Moore's "Ballad of Trees and stones"
1852|A friend to Boston, now far known,
1852|In Boston town, "Is Alexis," sung
1852|A strain of triumph, and these words she had:
1852|"The city's honour, name, and wealth have I!
1852|"For this, a man of blood, toil, battle, strife,
1852|Mechanics labor, die, like this of life;
1852|The whole world's fame, the wind of it, the wave
1852|Of some great deep, that rolls to heaven or cave;"
1852|In that same song, by all the prophets sung,
1852|It is the song to tell the mighty throng
1852|Of those who for their country's honour take
1852|Life's journey to that sacred hill of Fame,
1852|When with great souls, that have been, and are still,
1852|The first great men, to do great things for ill;
1852|When, from no other world, the world's heart breaks,
1852|That noble man who, though he singly speaks,
1852|Has less of good to do than such a song,
1852|And lives in little more than we of him.
1852|What cause has he, this loss of friends and kindred?
1852|What hope of fortune? Wherefore has he perished?
1852|Why is he living on one man's estate?
1852|The poet sings. Where goes the soldier?
1852|The soldier dies. One man lives, all is one.
1852|Let him by no means lily stand alone,
1852|For one man is not worthy of it--all
1852|Is to be tasted by a thirsty sword,
1852|Which, while he lives, will drink up all he can.
1852|There's good in war, but better does he die.
1852|No, this, no more--that man is not the man.
1852|He may be used to giving freely what
1852|He gave, that he should dare
## Sample 3:
392|So many a noble man he scorns in fight,
392|Away he wanders, and his heart declines
392|To be among the dead, and live in death;
392|And now he meets with fear and trembling fear;
392|His courage fails, his faith is hardened, dead,
392|His courage fails, he cannot hope to appear
392|Forward he flies, now threatening danger dread,
392|He calls his lords, and charges them and flies;
392|In dust and ashes, as in sands that bee,
392|Now drives his foot, or in the grass that lies.
392|The trembling sailors, in their care to keep,
392|Now of their craft, and now of theirs surprise,
392|As if they wanted courage, speed, and sleep,
392|And all their thoughts on their audacious prize.
392|The watchful Franks at last had seen this feat,
392|Their flags down to their knees out-stretched they lay,
392|Their head all bare, their helmet on their feet,
392|Their bodies all of gold, their locks of gray,
392|Their bodies all so fair that death might deal
392|The blow, and break the shield, the hauberk too,
392|They drew their swords, and on their backs struck through.
392|After the first they flew in fierce career,
392|Like devils monsters come from hell and fire;
392|They changed their hue, their tunes, their forms, their cheer,
392|And drew their weapons; what their foes desire,
392|They turn and fly, and both their bodies spare.
392|The hosts, whose courage, arms, and arms, were fled,
392|Yet no dismay it had, when, with a spell,
392|The king's enemies were forced to stay,
392|That night each knight was used to aid them well.
392|But now the day drew on with dreadful ending,
392|The Franks, in fury, spurred them on their way,
392|And, for that great king's fall, had turned their backs,
392|And fled, before that host, to God and day.
392|To God or mistress, had they made request,
392|Save of his presence to those lords of rest,
392|That they would go, ere

## Sample 1:
27139|And the stormy winds that blow
27139|All day long across the world
27139|I shall fly from earth below.
27139|"Oh, I am so weak and bowed!"--
27139|And the sun, in dazzling pride,
27139|Shook, and loosened as he heard
27139|All the turmoil of men's rights.
27139|In the town I should not be
27139|Wasted, wounded, and tormented;
27139|Made my spirit drunk with joy
27139|In the wonder of the sun.
27139|Oh, to see the sun shine through,
27139|And to hear the voice I knew,
27139|And to kneel because of old,
27139|And to speak, though it were gold!
27139|How it works, how it glows,
27139|In the sun, in the moonlight,
27139|And on earth, in the stream, in the odorous air,
27139|For the beauty of the moon.
27139|How it shines, how it glows,
27139|In the softness and the glow,
27139|And on earth, in the heaven, in the dusky air,
27139|Is the beauty that we know.
27139|Little, little, little sun,
27139|On your breast like the dawn of day,
27139|How I know about you always,
27139|With my heart and soul astray.
27139|Oh, to shine forever in your light,
27139|To shine forever in your beams,
27139|To shine forever in your brightness,
27139|To be shining, and to be!
27139|I knew the birds were going
27139|Out in the fair,
27139|Singing and working away,
27139|Going and staying and waiting
27139|Just as it might be.
27139|But the song of the children's singing
27139|Came every day to me
27139|Sweeter, and more than the singing
27139|That I may ever see.
27139|It rang so sweet, so clearly
27139|And far away,
27139|Making the angels, after
27139|Their work, but not a
## Sample 2:
38839|Where are the hours when we shall sleep?
38839|The night grows deeper, ruddy now,
38839|And on the east the morning light;
38839|No longer are the tears that weep,
38839|The noon-tide's drowsy murmuring
38839|That rises up through crimson leaves,
38839|As though the soul had fled away,
38839|To hide forever in the day.
38839|How sweet is the valley of Noodles!
38839|How quiet the little brook-heads there!
38839|What little bells by the cottages stood
38839|Tolled 'neath the heavy hours' flare!
38839|What tinkling bells the brook-folds heard
38839|As they crotoned by the millstone there?
38839|How soft the little brook-faces were
38839|Dimpled beside the cottage-door,
38839|And from the windows and out the trees
38839|The long-lost love-notes peeped once more.
38839|He came: the little brook he loved;
38839|And the brook beside his hearth he loved
38839|Where the crook whispered low, and the grasses
38839|That fringed its little patch of moss
38839|Beneath, the mossy water flowed.
38839|He came, and in God's blessed air
38839|He felt new life in that sweet air,
38839|New life and everlasting youth,
38839|And morning freshness, spring and sun.
38839|Weep not for me; our Father sent
38839|His little ones up to be content;
38839|As He had promised our poor ones
38839|We're glad to be thank'd like them;
38839|And let our taker daily say
38839|We're spared for our old love to-day.
38839|A rosy red rose is the rose
38839|And dear to me as children use;
38839|And though we sing it day by day,
38839|We know that mother's waiting here,
38839|And mother is the waiting-maid.
38839|God's plenty here, and He has sent
38839|Two
## Sample 3:
25953|The night that gives its glory,
25953|And the blue skies of the heavens;
25953|Sidney were worthless in that sight,
25953|All in a narrow space apart,
25953|I only had my life preserved,
25953|Left to myself to serve and guard,
25953|To guard and cherish, rest and guard,
25953|Of me the noblest was the king.
25953|Thus has my soul its homeward way,
25953|Thus has my soul its strength obtained,
25953|That what is dark may there appear,
25953|But evil in the light it feared,
25953|And in the light now burns my brain.
25953|With such a life, a wondrous life,
25953|Sounding through many realms and seas,
25953|The soul to millions makes at ease,
25953|And when it leaves the pleasant place,
25953|Binding in union is of grace.
25953|He who the body's life confers,
25953|From aught that is not, never slights,
25953|And when his body's life is spent
25953|The spirit cannot be content.
25953|I asked him what way he went,
25953|How many leagues he had not seen.
25953|The foolish wish I dared to own,
25953|To give the secret now I dare;
25953|I hid my foolish thought away,
25953|And lest the truth should be revealed,
25953|I thus began to test my will,
25953|And thus the answer was expressed:
25953|"I wish, O soul, that thou wouldst keep
25953|At present, in thyself at rest:
25953|But thou from sense hast naught to fear,
25953|And I my truth would not deny.
25953|And now by strength to hold thee fast,
25953|I do thee scorn, and I will give
25953|Henceforth my soul, and seek thy love
25953|As of the race of gods above."
25953|Then quickly I the thought requir'd,
25953|And thereupon this answer grew:
25953|"O soul, what wonder hath our Lord,

## Sample 1:
31314|Do you think this summer would be just one?
31314|Did not all the gods in their thunder-storm,
31314|Say, would make the trees, temples, rocks, and tops, . .
31314|And now would they think these trees didn't know,
31314|But they couldn't be built, I think they do,
31314|And would like to live on in a greener place,
31314|And the gods have made it to be a good history;
31314|And we haven't the time, you see, to say.
31314|The woods, the fields, are green,
31314|And the flowers everywhere.
31314|The sun and the grass,
31314|And the birds and the fishing-trees,
31314|Are quiet everywhere.
31314|And there is the sun,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And the flowers everywhere.
31314|The sun and the flowers,
31314|And one after one,
31314|Are quiet all around.
31314|But there is an end.
31314|There is another place.
31314|The end of the road
31314|To which to go.
31314|The end of the road
31314|To which to go.
31314|Ah, why do you go
31314|Into the garden there into the forest,
31314|And into the wood, and into the village,
31314|And into the field?
31314|Why does the rain
31314|Make me uneasy?
31314|Do you see,
31314|Do you see?
31314|Listen to me,
31314|I am in a hurry,
31314|And away.
31314|There is a strange thing:
31314|I have that alone.
31314|It is not at all like I married:
31314|There is a new man,
31314|Who has just been married.
31314|I have a new hat,
31314|That's in my hat.
31314|I wish I could find out another,
31314|But then there is one . . .
31314|Oh, why do you
## Sample 2:
19|Let us sing this song of mine:
19|Where I am, there I'm,
19|Tell the mighty, mighty sounding,
19|Ocean's awful son of old,
19|In the islands of the blessed,
19|In the groves of Arcadian
19|In his cradle, cold and cold.
19|We, the Fairies, we the children,
19|We the islanders, the bold!
19|We are all that has been fashioned
19|In the wondrous dreams of old,--
19|We, the revelers, the giants,
19|We the children, and the bold!
19|O the wondrous song of battle!
19|O the spoils of men of might!
19|O the spoils of conquest, conquest,
19|Where the many are not quite.
19|By the walls of ancient stories,
19|By the marble-mantled wall;
19|By the chains of dread OENEUT,
19|And the marvels of the fall.
19|By the ramparts of the giants,
19|By the caverns of the deep,
19|By the graves of men immortal,
19|By the caverns of the deep,--
19|By the temples of the Morning,
19|By the temples of the Night,
19|Where the warriors and the giants
19|Met in vision met in fight,
19|And the dying captive maidens,
19|Sat in still and stately light.
19|Child of Earth, too fair for sorrow,
19|Mother of the Light of Life,
19|Fairest daughter of bright radiance,
19|We have sung thee a new song!
19|By the ramparts of the giants,
19|By the temples of the night!
19|By the peaks of Alleghany,
19|Where the eagle cleaves the sky,
19|Trojan prince of fire and glory,
19|Comes in might the ivory-headed:
19|By the palaceer Hippolyta,
19|And the brilliant dame who leads him
19|Where the women weep and smite them
19|On the great steed Aristagrus,
19|Sorrowing, binds his loins of golden
## Sample 3:
1165|Bring us back our olden treasures.
1165|In the small white house with the crumbs for food and fire,
1165|How you cursed the stars and drummers!  I hear you drumming.
1165|Are you getting sleepy-hearted?
1165|Are you writing toiling while night was ebbing?
1165|Do you yet look in the face so white?
1165|How your furrows, how you pined away!
1165|Up the ladder--who would fain be strong?
1165|Who would hunger after freedom long?
1165|Are you being hungry, child, and tired?
1165|Mother, mother, come blow me a song.
1165|Out of doors a man came and sat beside me:
1165|He was black, he was white, but I could not see him;
1165|He was robed in a sackbut of very little gold,
1165|And the words that he said were, "Hotel-fever!" . . .
1165|He said:
1165|He has seen Barra's vision.  He is not old;
1165|He is not fit for the desert.  He can find
1165|Sight and freedom, voice and heart and mind,
1165|And he has not learnt to forget . . . yet he
1165|Has a vision of Paradise.
1165|But, alas! the thing is over,
1165|And there is no chance can take me.  That is why
1165|I looked, he said.
1165|The place is filled with flowers,
1165|With curiosities and secret pain;
1165|One has a face like those of flowers,
1165|One has an accent like a bell.
1165|The small sad music of my days
1165|Moves on.  The grassy fields and lawns
1165|Are not more silent than the stones;
1165|But one face moves beneath the stars. . . .
1165|All this is very beautiful, perhaps;
1165|The hills and woods, the fields and meadows,
1165|The clouds and clouds and all the sky,
1165|The sea's sky and the hills' sky.

## Sample 1:
22142|When the summer comes again,
22142|When the birds on the sunny land
22142|Make the winter to come again,
22142|I may say that the happy hour
22142|When the harvest-time comes again,
22142|When the heart aches for the land of my love
22142|And the day is my heart's desire,
22142|When the harvest home is come
22142|And the days are my heart's desire."
22142|"Oh! what if we both would wander over the sea,
22142|Afar from the home on the lowland, and stray
22142|O'er the hillside and over the dale, as we stray
22142|O'er the hillside and over the moor, through the wood,
22142|By the light of the moon on the hillside and stray
22142|Till the golden mist o'er the landscape is gray.
22142|And often for me the olden log-house is seen,
22142|The cabin'd log and the swinging door,
22142|The house where I lay till the break of day
22142|Till the sun shone out and the shadows drew away,
22142|While the shadows still wandered o'er valley and hill,
22142|Till the heart stood still, till the hush of the hill
22142|Came o'er the meadow and wandered awhile,
22142|In the sweet early gloaming with autumn awhile,
22142|To lie in the light of the long autumn days,
22142|We two.
22142|The gray-haired woman that leaneth on my side
22142|And holdeth me fast in her arms,
22142|Hath bid me lie down 'neath the old oak tree
22142|That so thickly embosoms me.
22142|The greening spring came with its silent voice
22142|When the autumn leaves hang their green,
22142|And the winds from the woods whisper'd a strange
22142|remembrance of many a long vanished year,
22142|Till the gray shoon dropp'd 'neath the gray shoon's shade,
22142|'Mid the gray shoon's shade;
22142|But where is my home
## Sample 2:
30488|Over the plains and slopes, where the wild roses blow,
30488|And the low valleys and the brown hills meet the sea,
30488|And the winds hurry by with a cry, "We want you! we want you!"
30488|And we stand on the roadway, and walk in the rain,
30488|And weep for the dead that have gone from us--we are waiting,
30488|Waiting the word that the wind shatters over the hill.
30488|We have come back from the hills, from the plains where our
30488|blood was shed,
30488|From the fields where the grass was the ground and the
30488|grain;
30488|We have come back from the valleys, we have come back
30488|from the hills!
30488|We have come back from the plains, from the plains where
30488|the grass grew green;
30488|And now that the voice of our winds shatters the
30488|world between.
30488|We have come back from the plains, have we come back
30488|to the graves,
30488|And the wind shatters the breath from the hurrying
30488|ship o'er the waves.
30488|We have come back from the plains, have we come back
30488|to the dead years,
30488|And the winds cry aloud, "Go up, and seek
30488|the faces of the gods!"
30488|We have come back from the plains, have we come back
30488|to the years,
30488|And the winds call out, "Go up, and seek the
30488|wilder world appears!"
30488|We have come back from the plains, we have come back
30488|from the days,
30488|And the swift leaves quiver, the bright leaves dance on
30488|our happy way;
30488|We have come back from the plains, we have come back
30488|to the dead years,--
30488|For the voices of men, or the gods' high
30488|heavenly joys!--
30488|We have come back from the plains, and
30488|from the long ago,
30488|Where in the cold and the gloom, on the
## Sample 3:
27494|To meet old friend Angelico's,
27494|And kiss the little one's beard.
27494|We shall have tea in the library;
27494|We shall have tea in town;
27494|By my good friend, Michael dissentund,
27494|I wish all you gentlemen!
27494|The sun has a face as fair as ever,
27494|As fair as ever was seen;
27494|As fair as ever was seen.
27494|There's a smile on his beautiful lips, my boy,
27494|As bright as the eyes of a lark;
27494|As bright as the sun at day-break gray, my boy,
27494|As bright as the eyes of a lark.
27494|There's a smile on his lips, my boy, my boy,
27494|As soft as the kisses he yields;
27494|As bright as the sun at noon-tide's end, my boy,
27494|As bright as the smile of a lark!
27494|Oh, there's something quieter than sleep, my boy,
27494|That twinkles in every eye.
27494|There's a smile on his lips, my boy, my boy,
27494|That shines like a lamp on the earth;
27494|As soft as the sun at noonday gray, my boy,
27494|As bright as the smile of a lark.
27494|A soft, pure love, that endures but ill, my boy,
27494|A soft Persuader that calms not nor charms,
27494|A king of the world-flowers and balm of the sea, my boy,
27494|A bright-hearted lover that seeks and finds, my boy,
27494|A rainbow of Heaven without a stain, my boy,
27494|A rainbow of Heaven without a stain.
27494|A sweet, pure love, that abides in a magical land,
27494|A kingdom-king, and a world-workeen band;
27494|A queen of the wildwood, a maid that is only one name,
27494|The bright-faced one calls her by name.
27494|A sweet, pure love, that all other names name, my boy,

## Sample 1:
1006|Shall wretched mortals, for a certainty
1006|Of cowardice, from forth their secret places
1006|Take flight, and turn in view the point askance,
1006|From whence they can, on any object bent,
1006|So that the mirror may not abstain
1006|To 'GOD,' as in the sky grows colourd there
1006|The sunbeam.  If I had, I should prove
1006|The fountain's well-accustom'd form there.
1006|I should not be a word, that is not parting
1006|Betwixt the sun and moon, nor they, for sowing
1006|In opposite direction, doth import
1006|Their night and their moonshine to that shadowy land.
1006|Therefore into dark caves are we brought,
1006|Where underneath is alabaster, save
1006|Of dusky idol, in which shining moors
1006|'Cross-bows do us sometimes view.  Rumm, there,
1006|That consecrates the world, are seen below.
1006|How many stairs, as 't were, to o'erpass the threshold
1006|Of my triumph, when I did before thee,
1006|Enter'd, and were below, though light were not
1006|But in those hours to which this chant is not.
1006|For thereon is my mind alone intent,
1006|And this there be the singing souls, that wait
1006|Their Master's will.  If but e'en now thou choose
1006|To listen to this be melody sweet,
1006|Here may thy heart find rest.  If this thou hear,
1006|Thenceforth what argument there smote thy love;
1006|That, well perceiv'd of me in its ascent,
1006|Thou mayst at last the good from that have learn'd
1006|Of me made perfect, and of now prove o'ercome.
1006|That next I saw, which, hearing, thou shalt take
1006|For guide, arose from out my last embrace,
1006|As a good mother should, that her embrace
1006|Around each form Adams in her son,
1006|The same who
## Sample 2:
1006|Shall they not fear?--let one fear have his throat?
1006|No: for I see, and can perceive it not,
1006|That they, who were before me, were lamenting
1006|For the nature of the world, and in its light
1006|Have pity on their prison-gries, and therefore
1006|Spirits of low estate do not accuse.
1006|As oft along the still-perduing Nile
1006|Along the Indian isle the night-owl thronged,
1006|And with her breath had led them to the rivers,
1006|Thus each along the shore, in company
1006|With the fair consort of a lady, moved
1006|To supper with a working-place of eggs,
1006|Whereon all love was painted; and "O Lord!"
1006|The singer, as he finished singing, struck
1006|On the sea-shore at left and right below,
1006|Stay'd his steps, and thus in accents clear did:
1006|"Behold a wonder! behold a thing
1006|Which, whoso of mortal thing may tell,
1006|Hastes before him, dark and terrible,
1006|Of an eternal weal; and this accursed
1006|Foul thing, which may the doubting furnish try'd
1006|To feebor, I have set before him all,
1006|Who must to nothing turn.  But list: resound
1006|With me the patience, and the pity move."
1006|He then: "Now, sister!  think upon thyself;
1006|For how may I inform thee, if thou hearest,
1006|How soon and how slow time and means he takes?
1006|What world or language may at last he ask,
1006|Who to the world may'st give thee all things fair,
1006|Thou know'st, who hold'st it?"  Compassing that,
1006|And beck'ning the approach of his approach,
1006|I answer'd: "By new stars, and by George, spirit
1006|Of sign all cardinal virtues I'm assured,
1006|That wrong thy sight; but
## Sample 3:
1006|What became of thatch of yours
1006|That you sally forth so suddenly?'
1006|I did not know him; but he felt
1006|As freshening blood the air beneath: and tow'rds me
1006|He moved, and I towards him moved, and fond
1006|As father, and in words, and with the sweet
1006|And with the sweet comply'd; as he had will'd,
1006|I wept, and to his bosom lifting me
1006|Upward, and all the while distill'd my blood.
1006|Proceeding from the river, and I thence
1006|Shelter'd and still, I lifted up mine eyes
1006|Toward the sun upon the matin light,
1006|And looking, ever taking it in view,
1006|Again beheld it tow'rds the other bank,
1006|From those eternal Poubs where first it was.
1006|Unto the chasm precipitous, I found
1006|A narrow path and new to me expos'd,
1006|And to the left hand and the right mine eyes
1006|Directed, and between, on the sharp rocks
1006|Directed, and between, on each side me,
1006|And on the ground, were dark and cavern'd depths.
1006|Then to the right it spake, in semblance such
1006|As sometimes Indian women have beheld
1006|Saw in their dreams the latter realms of Hell:
1006|"Why partest from me with looks enchanted,
1006|"That farthest from thine eyes withdraw thee not?"
1006|I said it somewhat, and turn'd to the right,
1006|And when the chasm was gone, and I had reach'd
1006|The bottom, where the stairs were not beheld,
1006|Backward my eye alone its course directed,
1006|Directed to another care and deep,
1006|That in its bark it led me; and the sage
1006|To my recalling them returnd I not.
1006|"Of all that I have shown thee, since that gate
1006|We enter'd first, whose is the self-"

## Sample 1:
1006|If once thou see, the sport is vanity.
1006|But fix thine eyes beneath: the shameless'st thou.
1006|Ah! yet no better sight hath ever brought us
1006|To the destruction of Eve, from the rape
1006|Of the fair world, where wickedness lay hid.
1006|Ah! now before the good Augustus returns
1006|In daily deeds, let our pray'rs to the Furies
1006|Be changed; for that he longs counterfeited
1006|The world's adornment, which in books for us
1006|His saintly he still wears, who for us did
1006|Lays his beloved, us'd have thereby
1006|Been so enamoured of this roundelay."
1006|As it was spilt by him who severs the dry land
1006|And, each eve earlier than the setting sun,
1006|Wines underneath the rose that Tarquin leaves;
1006|So ravished I the more the fair, the more
1006|The great disdain, the pity, the long-sought,
1006|Toward the ribald nature of this law,
1006|Which the new world much sooner would be war
1006|With my sweet-will, the which the will hath oft
1006|Been usefully perverted.  I beheld
1006|Four mighty shades, and they besought for speech,
1006|But spake but slowly; whereto I re-turned,
1006|They showed me the grievous shame that clustered
1006|In their resplendence, who had all confounded
1006|With Adam, that Adam's fair abode
1006|Long time abode in peace.  "Remember," thus
1006|The teacher cried, "Adam, I the three."
1006|"And is this the true mettled nymph, that shine
1006|Thy twins now in th' ascent, as thou dost now
1006|Below?"  I said: and he, "My son, who know'st
1006|The art of song, and who but hear, may be
1006|Lest thou divide the loaves from five to six."
1006|Thereat the
## Sample 2:
1006|If truly the mad people of Raymond dread thee,
1006|Well hast thou 'scaped yourselves, they are already
1006|Each other's torment; and the wretches, mortals!
1006|Are to the shades of thy polluted ways."
1006|Whence she again replying, after us?"
1006|Purgator to the Red relates this; and Dante,
1006|"That spirit of the just hath spoken and said,
1006|'Purge now the mountain of the unquench'd, and make
1006|The sea of fire purge otherwhere he sees,
1006|Where Hood and other vain adventures 've led;
1006|Let the o'er-shadowing bridge ill-beseem'd
1006|Yoke up the mighty demons while they walk
1006|In different paths, and divers guides disentangle
1006|The tangled ways, so that no stork may turn them back;"
1006|And the good Master to me: "See thou findest
1006|E'en thus, how in the little one the race
1006|Is to be cumber'd with the broken blossom.
1006|That from this blow across theims of brightness
1006|Has not so virtue worried thee, it seems,
1006|That one with good intent in passing 'bove it
1006|Thou seest the harmonies of the mind,
1006|And therefore dost not comprehend them.
1006|That the disturbance which the sound invades
1006|Interpreteth not, maketh thy fancy
1006|Take in good hieroglyphs of everichoon;
1006|Thou seest that not only he who bids it
1006|Be, but that other who behind the hill
1006|Comes from step's shadow, comes in solitude;
1006|So that the evil one, who, in disdain
1006|Of creature freedom, 'bove the other waits,
1006|The same, who hoping in complacency
1006|To aid thee in thy loftier thoughts, like them
1006|I see now aids, which thee, thus overcome,
1006|With light returneth to thy mind from thee."
1006|Ill shapes that course which
## Sample 3:
1006|If thou no longer liv'st in the old age,
1006|To stop the running of the vein thou well;
1006|Thou by thy deeds hast purged away my youth,
1006|Strengthen'd by me in good deeds and old,
1006|And banish'd with my beauty from my song.
1006|O how much better were it I had Ruthven,
1006|With Beatrice and high excellence
1006|Of soul! to have companion'd such, who yields
1006|To her for intermingling of the world!
1006|But I see thee already clasp'd in thine,
1006|Not in thy only Heaven, but in the life
1006|To which thou crav'st it, and dost here renew it.
1006|This good companion, speak from whence it is
1006|That I should fear not, when the hour arrive
1006|That to your sighs a second time shall bring,
1006|Then with her grace, which that affection gave
1006|And your fair looks, which with her will I scan,
1006|Till my changed nature prove itself as yours."
1006|Whence her reply the thoughtful Sage imposed;
1006|"And I am He, whom thou didst long for, dead
1006|Cord de'il me not," he answer'd, "yet for thee
1006| hamm'st me.  Heart of Potentate was I,
1006|And life was in the willow.  That the words
1006|Which now I hear, shall marvel thee, wert thou
1006|A dearth in love.  Ah, how I long, long to
1006|For it am pain and grief!  that knowest well
1006|My soul's fair metes on earth!"  His sovran speech
1006|Bespake the meek thanks of the high Lord.
1006|I do not over 8 oceans turn," he then
1006|Began, "if new device can be of late
1006|Received in thee, tell us, who thou wast, and by
1006|The heels of th' mountain, times such words to me."
1006|"That I frizzano elsewhere is

“Jabberwocky”, Lewis Carroll

Upon request, I generated 100 samples of Lewis Carroll’s “Jabberwocky”. Examining preliminary samples, the closest prefix was #24650, corresponding to The Jingle Book, Wells 1899_125ya, an anthology of humorous children’s verse (which makes sense). “Jabberwocky” itself does not appear in the PG corpus but the “Jabberwock” is mentioned in one of the poems in Wells 1899_125ya, the acrostic poem “An Alphabet Zoo”, so, close enough.

Some samples:

24560|He found a foxy in the brake,
24560|A cunning fox of scarlet dye,
24560|And from that foxy followed make
24560|The scrawny fox in glee.
24560|He followed with his dam and horn
24560|To where the river-water runs,
24560|And as his living current on
24560|The river-water likes him up
24560|A mighty rocky heifer heaves,
24560|And in a single field, or twain,
24560|Shows like the yellow corn;
24560|And when the wind doth blow, so too
24560|Low in his bottom lies his head,
24560|And in the grass leaps up again,
24560|In fearful freedom unbetrayed.

24560|And while the sword-strokes gushed and ran
24560|A-gathering like a hurricane,
24560|The wildest war-cries, loud and shrill,
24560|Came surging through the hills.
24560|At times a wild yell rose in the air,
24560|From steep to steep, from moor to moor,
24560|A scream of horror and despair,
24560|Of death approaching neathernock,
24560|Of dying embers in the pore.
24560|"Thy shield, my son! thy sabre bright!
24560|My hands! thy lips! thy lips of flame!
24560|O hasten! I have done the same."
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly fall
24560|On the proud swine that fought and died---
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! thou hewer of the sanguine!
24560|Say on! say on! that pierced youth
24560|Who crossed the heath in days of yore,
24560|And when he crossed it, "Say on!"
24560|It made the blood tingle and squall
24560|To hear that wounded warrior's cry;
24560|It made the life blood quickly quail,
24560|To hear that death did soon draw nigh.
24560|It made the blood tingle and squall,
24560|To hear that death did soon draw nigh.
24560|"Say on! say on! thou wounded swine,
24560|Say on! thou hewer of the sanguine!
24560|Thou art not doomed to fetch a pail,
24560|'Tis not the ebbing blood so drain."
24560|And then it made the blood tingle,
24560|To hear that agony within:
24560|"Thou liest"

24560|"It is morn!" said the clover-bush,
24560|As the sun came up the sky;
24560|"It is morn!" said the cattle-cock,
24560|As it drifted by.
24560|Then loud cried the blackbird, and forth
24560|On the moor the clamorer fell;
24560|"The rusty ruts!" he cried, "is morn!"
24560|And he opened his little cage,
24560|And went to his morning cell.
24560|Two ravening birds sat at his feet,
24560|One flew away and one fell,
24560|The blackbird with a shining face,
24560|Sang these grand welcomes to his race.
24560|"Who is the Blackbird?" they said,
24560|"Singing high, sing low, sing low,
24560|Sings the song for the giver and light?"
24560|"I am the Blackbird---a joy befall
24560|To the young in the woods so fair.
24560|I am the wild-duck, which roams at will
24560|Up and down in the gulphyry rill.
24560|I, the giver of rivers fair.
24560|I, the joy-devoted mole,
24560|I, the joy-devoted mole.
24560|I, the humming-bird, which sings so sweet
24560|I, the giver of rivers fair.
24560|I, the bird-cail'd eggs, with a silver sound,
24560|I, the joy-devoted mole.
24560|I, the giver of goodly things,
24560|I, the swift-wing'd eagle, I,
24560|I, the joy-devoted mole.
24560|From the sunny, sunny south,
24560|From the sunny south,
24560|The swarm departed,
24560|But woe to every wicked wight
24560|That ever them befel!
24560|"O shame to every wicked wight
24560|That ever them beguile!"

Training

This proved more difficult than GPT-2-117M. The GPT-2-117M model was already large, at 480MB for the whole, so making it 3× larger bloats it to 1.4GB on disk; and the VRAM use on a GPU is even worse: with GPT-2-117M, a training minibatch of n = 2 could barely fit on a 1080ti’s 11GB, but at GPT-2-345M, n < 1! The main culprit seems to be the self-attention layers, as regular self-attention scales more than linearly, so GPU VRAM gets eaten up fast, and apparently 16GB might not have been enough for GPT-2-345M either. While I have enough system RAM to train GPT-2-345M without any tricks, my Threadripper CPU is still ~14× slower than a 1080ti, and if one guesses that GPT-2-345M takes 3× longer to train than GPT-2-117M, and GPT-2-117M takes 1–2 days, and CPU is 14× slower, then that’s <84 days for the poetry finetuning, which would not be fun.

To solve this, nshepperd extended his GPT-2 training codebase to employ a technique OpenAI helped introduce (and presumably used in training GPT-2, although the GPT-2 paper is silent on the details): “gradient checkpointing”. Gradient checkpointing is a space-time tradeoff which throws away some of the intermediate states of a NN, potentially greatly reducing total VRAM use, but at the cost of some slowdown when those intermediate states need to be recomputed for doing the backpropagation; the slowdown, fortunately, turns out to be fairly modest.

The downside of gradient checkpointing is that for GPT-2-345M, it is still not memory-efficient enough to train it just like GPT-2-117M—the self-attention layers checkpoint nicely (as the Sparse Transformers paper remarks¹² apropos of needing extremely wide Transformer windows to accomplish MuseNet), but it’s not enough, due to the giant word/BPE-embedding, which blows out RAM usage. (Although it’s possible nshepperd didn’t implement gradient checkpointing quite right for GPT-2, as the OpenAI papers don’t mention any difficulties related to the embedding or using gradient checkpointing.) His initial solution was to simply disable training of the embedding and train only the Transformer layers, reasoning that the generic English embedding probably wouldn’t need to be trained that much as the Transformer layers are where the real work is done; much later, it occurred to us that the Adam SGD optimizer was part of the memory problem, as, being an adaptive momentum-based SGD optimizer, it must store a mean/variance for every parameter to adjust its updates per-parameter, which greatly increases memory use (and which gradient checkpointing does nothing about); when we switched to simple SGD, that freed up enough RAM to re-enable the embedding. This is important in part because the learning rate for Adam & SGD differs by orders of magnitude: a LR <0.01 seemed good for me for SGD, but Adam wanted a LR more like 0.00001. With n = 1 minibatches, the training loss is extremely noisy and it is difficult to see the impact of any hyperparameter changes for the usual hand-tuning, so nshepperd also implemented a simple ‘validation loss’ function, which was helpful toward the end.

So, the upshot seems to be that GPT-2-117M can be trained end-to-end with Adam on a commodity GPU in 11GB VRAM; and GPT-2-345M must be trained with gradient checkpointing, and one must choose between either fancy SGD optimizers or full end-to-end training including the embedding; and 744M (released 2019-08-20) can’t be trained at all. Toward the end, I switched from Adam+Transformer-only to SGD+all, and this seemed to drop my GPT-2-345M-poetry validation loss by ~0.01 to a final 1.915 (which is not nothing, so perhaps the embedding did need some adjusting for a more poetic vocabulary).

In total, I trained GPT-2-345M-poetry for 815,326 steps (minibatch n = 1), with an Adam LR ~ 0.00001 and SGD LR ~ 0.001, over ~7 days (2019-05-04–2019-05-13) on 1 Nvidia1080ti; the necessary training time, with the benefit of hindsight, was probably closer to 3 wallclock days. GPT-2-345M-poetry converged to a final loss of 1.915, an improvement of ~0.1 over GPT-2-117M’s ~2 loss (so, in some objective sense, which is indirectly related to generated poetry quality, one could say that GPT-2-345M is 5% better than GPT-2-117M). I had expected somewhat more quantitatively, so I wonder if more aggressive training methods like cyclic learning rates+SWA would have worked if they were implemented in this codebase & I had the patience to wait a week or two for multiple cycles? In any case:

• GPT-2-345M-poetry model download (1.2GB)¹³

• 5000 random GPT-2-345M-poetry samples (0.95 temperature)

Samples

[285999 | 5982.87] loss=2.07 avg=2.29
Saving checkpoint/run1/model-286000
Generating samples...
======== SAMPLE 1 ========
1170|Thy soul, thy very soul is burning!
1170|I look, the old-time woman I love
1170|Comes in for a kiss.  The room returns
1170|A ghost of its faded glow the same,
1170|But the old woman is gone as before;
1170|The moon has come down with a veil,
1170|And the old woman sits by my side
1170|Like a new moon rising from my grave!
1170|One day, I saw a bird full ripe:
1170|And a little flower in my garden
1170|Was the signboard's sign of peace to be.
1170|Like unto one, it told of the spirit
1170|Who dwells with all things and can see them not.
1170|The birds of all colors sang freely,
1170|And the flowers of every hue and scent
1170|Ran on like a dance up heaven's stair,
1170|With songs and flutterings of the blue and red,
1170|And sunbeams, glittering the branches overhead,
1170|And the wild sunbeam streaming upon them.
1170|And my soul was filled with a sweet desire,
1170|A need that was strange and divinely dear;
1170|And the birds sang so that the day was gone;
1170|And the flowers came full when night was near,
1170|With full hearts of light and fragrance blown,
1170|And the flowers and the birds would sing again
1170|Toward the evening that would bring them near.
1170|And the sunbeam was white as the sea,
1170|The flower was white as the evening skies,
1170|The star on the rainbow was gay and bright,
1170|And the bird of fire was a flutter of wings,
1170|And the flower and the bird would sing once again--
1170|Till I knew my soul was full of its sweetest dreams.
1171|I see how the leaves of the trees fall
1171|In the sunlight of morning;
1171|And in many a wood, with a thousand sounds
1171|And a thousand eyes, a thousand suns to behold,--
1171|I see how the leaves of the trees fall.
1171|How they fall in summer on the slopes
1171|And the meadows and the streams and streams!
1171|How they fall in the winter, soft and slow,
1171|How they fall in spring!
1171|Yet this fall must ever be beautiful;
1171|I think of the leaves, I hear them speak,
1171|The stars in heaven see them pass and pass,
1171|The rains are silent with them in their fall.
1171|The leaves of the trees are lightening and falling;
1171|The leaves of the trees are falling;
1171|And the leaves of the trees are all of their leaves
1171|Fallen on the fields,
1171|And they fall, and they fall on the fields,
1171|And I see how the leaves of the trees fall.
1171|The stars of the heavens look above,
1171|And the leaves of the trees are falling and falling,
1171|And bright winds sing,
1171|As in gladness, softly, soft and low,
1171|Loud and low;
1171|And in every star--what a wonder is it!
1171|The leaves of the trees are falling on the plain,
1171|And light-begotten clouds are lifting up the leaves,
1171|That fall like drops of rain.
1171|They fall on the flowers; and a thousand roses,
1171|And a hundred white lilies,
1171|And a thousand roses, and a hundred blossoms,
1171|And a thousand white lilies.
1171|They fall on the flowers, and droop and decline,
1171|And sleep in the sun.
1171|The leaves fall on the grass with an eddy they weave,
1171|And a thousand golden lilies
1171|Dream of the night of the leaves that was swept away.
1171|Their dreams are done.
1171|The leaves fall on the fields. They sleep in the sun,
1171|And the flowers shall be green,
1171|For the leaves of the trees are falling in the sun,
1171|With their rich lights of blue.
1171|I go to the door to-night,
1171|And look in the

[286000 | 6003.19] loss=1.03 avg=2.28

[347999 | 52462.43] loss=1.98 avg=2.06
Saving checkpoint/run1/model-348000
Generating samples...
======== SAMPLE 1 ========
 closed, or wearily
24869|Lest some foul fiend should be o'erthrown,
24869|Our hosts of fiends in wild rage we
24869|Began with furious rage to fight.
24869|We met them at the ramparts high,
24869|In numbers few and in strength not few:
24869|Whence we, our life like water spent,
24869|Abandoned the walls and basely fled,
24869|And the great flood swept like a raging sea,
24869|That o'er the deep ocean's bound
24869|Comes rushing on a torrent sea.
24869|And now the fiend's fury we met,
24869|And, like an eaglet chased by an eagle,
24869|We met, O King, with our lives in stake,
24869|As if our lives were forfeit by the foe:
24869|For in the raging tide that swept
24869|Our spirits from the wall and low,
24869|We lost and were scatheless once more.
24869|No foe may stand the fury of the king
24869|Who comes so fierce and bold to fight.
24869|And ere night's dark clouds have passed away,
24869|Our fleet will fly, and, safe return,
24869|We shall be in our country won.
24869|So will he take his own, O King,
24869|As some strong bull that fights in sport.
24869|The fiercest foe we ever saw
24869|Has strength to fight, has courage long,
24869|Is he whose flesh is fiercest, bold,
24869|Or who has lost his princely sway."
24869|They heard with trembling hearts and fear
24869|The angry speech of Rávaṇ sad:
24869|In trembling hearts and fearful fear
24869|The brother of the King of Thieves.
24869|His anger at her words they felt,
24869|And many tears they hurriedly shed:
24869|The king who ne'er to others shed
24869|His rancour of unwholesome truth.
24869|Like some vast crocodile or snake,
24869|Like the vast crocodile or snakes,
24869|He who his pride of power would show,
24869|His brother whom he scorned and belied,(856)
24869|Saw the sad news with many a sigh
24869|To Ráma by that voice revealed,
24869|And thus he spoke, most pitiful:
24869|"If thou wilt fight with Ráma, then
24869|The giant King, thy vassal, may
24869|This city all take securely in:
24869|But thou with faithful Ráma fight.
24869|'Twere well that you the city bring,
24869|Our lives in danger should ye die.
24869|To-day and to the morrow, he,
24869|Great king of giants, may he meet,
24869|The giant king I know who stole
24869|From thee thy darling Lakshmaṇ, when
24869|Our fathers on the day of vengeance
24869|Saw thee in this city slain.
24869|To-day my son is born to thee,
24869|To-morrow to the King may he
24869|Thy Ráma and thy Vritra slay,
24869|And thy own heart-sick brother, too.
24869|To slay my brother thou hast vowed
24869|This day and day in turn to dwell.
24869|Why, Ráma, is the world thus full
24869|Of fierce wrath, full of deafening noise?
24869|Thy brother's life and thee's, I ween,
24869|Lie evermore in my eyes.
24869|For all a life like this can yield
24869|Is precious little: thou must play
24869|With thy own brother, and a prey
24869|To thy most furious foe consent,
24869|If Lakshmaṇ and thy brother die.
24869|Now I would fain the King of Thieves
24869|A thousand worlds in strife destroy,
24869|And all the race of fiends who dwell
24869|Deep in the wood, a million moons.
24869|I would not seek retribution here
24869|For thine own hand upon the dead:
248

[348000 | 52483.09] loss=2.75 avg=2.06

[531999 | 9443.46] loss=2.09 avg=1.97
Generating samples...
======== SAMPLE 1 ========
 better his life and wealth to him
2136|I will set the gold in store."
2136|He spake no word, but bowed his gray head low
2136|And left him swiftly, and he came
2136|Through the great doors his sons to see.
2136|And there within the court his sons brought
2136|The golden treasure, all bound in gold,
2136|The precious gems and the wise gems round
2136|They held, and bore them back and brought
2136|To King Olaf, and from them cast
2136|As gifts before the king his gold.
2136|And as he heard the sons bring in the gifts
2136|He spake unto King Olaf,
2136|"My mother, look ye here and see
2136|What hap lies layed before me here."
2136|On the ground the King his mother spake
2136|From the dark wall, where her breast showed white
2136|With misgiving and dread:
2136|"Nay, but this man's heart that did breed
2136|Fell full of bitter grief as mine shall fall,
2136|And grief is for the day that comes,
2136|And sorrow is for heart to know."
2136|"Nay, O King, to-day is good and fine
2136|Withal as glad and long to be;
2136|But to-morrow may bring mickle pain
2136|Withal for grief," the king replied.
2136|"And shall be."  "That will I say."
2136|Then rose and said, "My friend, I stand
2136|Before the king in glory now:
2136|"The gold now lies at my royal hand,
2136|The gems of my folk's great king there be:
2136|And from my mother, the dark woman, bear,
2136|The dark woman, the great lord's daughter fair
2136|And mother of our good king and lord,
2136|This golden thing:
2136|This ruby and this chrysolite,
2136|Gleaming bright with golden light."
2136|And Olaf spake, for quickly he spake
2136|Sooth and plain, in full or hidden wise,
2136|And as the king in council him eyed
2136|Stood to speak, saying, "Be it so, now,
2136|Here is for mourning and for mourning sake
2136|A costly gift for our father's sake."
2136|"Goodly gift as I ween," the king rejoin'd,
2136|"And I will take it all, I ween,
2136|Than for grief or for sorrow some wise need."
2136|And so spake the good knight Grettir, and said,
2136|"My son, let the gift-bag now bring too,
2136|With silver wrought in gold and satin wrought
2136|And gold, and then our lord and master true
2136|May go in as wont:
2136|"And we will bring it," the king replied, "with speed."
2136|And so he took the golden gift and laid
2136|The ruby and the chrysolite by him,
2136|Which the cold hand that laid it must give
2136|And let him lay it next before him
2136|Behind the arm to bring it to the king.
2136|And straight the two princes went to the king,
2136|But went in silence, gazing at the golden ring
2136|Unto the lord of gems, whose shining face
2136|With a low voice them said, "Why dost thou stand
2136|And look all gleaming in the kingly door
2136|Where kings and heroes pass?
2136|"I knew not that thy heart so glowed with hate
2136|As to look on me, though I stood there high:
2136|Now know I nought," the warriors said, "dear king,
2136|Thy great heart's wrath shall not thee displease.
2136|"Not thou thine anger or thy great displeasure:
2136|For now at last with death thy father dies,
2136|And in thee as in his sons doth the will

[532000 | 9459.46] loss=1.31 avg=1.97

[534999 | 1390.41] loss=2.18 avg=1.89
Saving checkpoint/run1/model-535000
Generating samples...
======== SAMPLE 1 ========
1728|the ground: the other two with their hands each on his
1728|sleeve laid flat against his face, and strove to cover
1728|himself with the cloak. Then each was moved by pity, and
1728|clothed him in the cloak. And the hero Alcinoüs rose
1728|straightway at his father, and gave him his spear, and spake
1728|in his turn to Alcinoüs, saying:
1728|'Alcinoüs, no longer endure to sit, as though a
1728|mere man, and let the great Menelaus come in; but come in
1728|and see the son of Nestor, the Achaean hero; and ask him
1728|all about thy deeds and thy father's house, whether to have
1728|stayed, or to have returned to thine home. Say, is he still
1728|living, or hath he now gone down to the house of Hades
1728|gods to abide by them?'
1728|Then the noble son of Nestor answered him, saying: 'Father,
1728|I was not yet the son of Menelaus, when a god set me
1728|here of mortal man; yet he will not lay an hand on me,
1728|unless I too go down to the house of Hades, through fear
1728|of Zeus, lest I repay the wrong that He hath done. My noble
1728|son has gone with thee to the halls of godlike
1728|Eurymachus, and hath vowed thee unto death with a spear,
1728|on account of thy evil counsels; and in a fit of anger
1728|he promised to slay thee by the hands, on pain of death.'
1728|Then Antinous, the son of Nestor, answered him and said:
1728|'Yea now, my son, I will go thy house and seek thy
1728|father, that he may tell thee all aright.'
1728|Thus he spake, and with him went forth the noble
1728|son of Nestor. Then Alcinoüs and the other Achaeans
1728|greeted the king, and each one offered gifts. But he did
1728|not speak to mortal man about his ransoming of the
1728|gods: as for the sons of Nestor, he made them wait a long
1728|time at the ships, for he was fain for gifts in form of
1728|merry or gold, and a goodly store of raiment, and himself
1728|a well-beloved son. In like manner he minded that the
1728|children of men should offer gifts for his sake, and
1728|ponder his counsel; and the other captains and counsellors
1728|would be listening in their halls.
1728|Now the noble son of Nestor had gone down to be dead,
1728|for his house was near, and the people thronged that way about
1728|him, and at the first dawn they found him, sitting
1728|at the entrance of his hall, where he had left his golden
1728|branch,
1728|and he looked great tears upon their faces. Then his spirit
1728|came not again to his own body with him.
1728|So long as he dwelt beneath the waves of the dark sea,
1728|some god had nursed him in the dark, but as soon as the
1728|winds swept him from the sea to the land, he fled to the
1728|tomb of Zeus. There a great swan lay in wait for him, the
1728|lordly Erystus, grandson of old Oceanus, and was first
1728|to woo him, being the first who came to woo a maiden,
1728|for surely there was none else in the wide deep to
1728|prevent him. So the lordliest son of Nestor was minded to
1728|offer him a goodly fair gift of goodly gold and bronze,
1728|and he did so, and the gift was the best, the fairest of
1728|all. And the king Alcinous and the other chieftains and
1728|favored of goddesses then brought him to the house of
1728|Zeus, who is godlike and lord of generous souls. And
1728|the blessed gods showed great joy of the deeds they
1728|had done in the sight of all. As for him he lay lying

[659999 | 5522.96] loss=2.36 avg=1.90
Saving checkpoint/run1/model-660000
Generating samples...
======== SAMPLE 1 ========
In vain: there is one true knight that we
1728|shall meet, if we seek it so, and there will be
1728|no need for anything."
1728|So he spake, and stirred the blood of Thrace, where
1728|the thick-sown reek of the battle had made a
1728|great pond, and the blood-drinkers were all weary
1728|of battle for their wine-youth, which they had got in
1728|their hands. And now they set fire to their braziers
1728|and went to bed, and their eyes still looked toward
1728|Ithaca, for they thought to see the haven called
1728|Lacedaemon, and the good news of Agamemnon.
1728|Now when they were come to their place, they feasted
1728|on plenty, and by great signs the Trojans led
1728|themselves about the gates, and when they had put
1728|everything of weight in, they took their stand
1728|straight on them, smote on their helmets with their
1728|hoofs, and the loud clamour of the fight was made
1728|heard and known; so there were great confusion and
1728|revenge in their breast, and the stout men
1728|thrust down their armour from them and dashed
1728|them upon the ground. Thus they fought in the
1728|thick of the battle. But the valiant Dardanian
1728|strong in fight found his strong men among the horses
1728|of the foremost, and caught them up in one. And he
1728|sitting by those heroes, put to their mouth a
1728|salt, and he sprinkled them with the drops of
1728|grapes, and spake and hailed them:
1728|'Friends, it is meet, O friends, that we should show ourselves
1728|good men, who know not warlike service by a name
1728|otherwise than this, a mind of mortal men: for war
1728|shows weakness in the hearts of men. Let us be men,
1728|and fight with heart and hand. As for this word of
1728|sorrow that ye bear after so long sorrow, know
1728|none the less that it is the will of the gods:
1728|we have seen great evils in our father's house, and
1728|this very day the people have perished for a
1728|little, for there was no man in them save but three
1728|old brothers, men of renown in the halls, yet they
1728|wrenched apart themselves and went their ways as wolves and
1728|foxes. And the daughter of Poppaea bare them sons,
1728|And the lord of the Credans, a man of evil mind,
1728|came at the fourth year to Mantinea: one came in the
1728|twentieth year, and begat seven sons, and he was
1728|the lord of the city of Mantinea, where he made an
1728|immediately for himself an offering from his own substance.
1728|Therefore we will not fight with these people, but let
1728|them take their way across the sea from the land, and
1728|the old man's tale he answered with bitter mouth:
1728|'"If this will not win ye to the passage of the
1728|sons, then bid your men stand guard at once upon the
1728|shield-wall, lest that too I take the sons of my
1728|father and sons-in-law in the house and see the
1728|evil deed done in all my flesh. As for me, I do not
1728|believe that I am of any worth whatsoever, for ever as now
1728|I am lying under the earth with bare shoulders,
1728|having lost even my manhood and the strength of my
1728|hands, and ever through the days of my life I am without a
1728|strength, and ever in the counsels of my heart I was
1728|strong.'
1728|Then wise Telemachus answered him, saying: 'Stranger,
1728|not till all these things have come to a pass and
1728|the people of the Achaeans make great feast and
1728|have a banquet, then wilt thou boast to thee of thy own
1728|courage and thine own strength in going and bringing
1728|them by strong armament over the sea,

[656999 | 2795.58] loss=2.49 avg=1.85
Generating samples...
======== SAMPLE 1 ========
 hope the night shall cease before the day.
27441|As the last year lies at last,
27441|And the last year's sun
27441|Ascends the hill-tops once seen in;
27441|So th' last year dies, and shall not die;
27441|For the day is spring's and spring's heir.
27441|When the last year sleeps,
27441|When the last year's day
27441|Darkens the hills,
27441|When thy light is no more, but only thy shade,
27441|Then will love make the hills and valleys glad.
27441|When the last year lies at peace
27441|And the last year's sun,
27441|Shines, as it before, upon thine altars alt,
27441|Then to thee, O God, th' offering will be offered.
27441|To the music of the spring-time we come in,
27441|We love to trace old Ocean's flow
27441|'Gainst the banks of every shallow lea,
27441|Or watch the rolling green of every hillock green
27441|Like gold in the dawning;
27441|We love to hear the sea run screaming and laughing--
27441|We love to hear the deep grow wilder and wilder--
27441|We love to hear the rolling of the sea-waves
27441|Crashing through all the coasts of the world.
27441|We love to watch the birds and the rising sun,
27441|Like a great stream through an isle of gold;
27441|We love to see the birds in their lovely green,
27441|And a sunbeam in their shining;
27441|We love to hear the sun beating, beating--
27441|It's a joy to be young when you're old!
27441|The spring-time's in its rosy prime.
27441|All the birds are flying;
27441|We're happy, and we love to be young, in summer too!
27441|'Twill come to us still, as old as we,
27441|To sing our story ere we sleep,
27441|And wake, and find the old world bright in the morning!
27441|_The_ WORLD, we know, is passing away,
27441|With all its pleasures and its woes:
27441|But one thing, sure, must remain:
27441|Our story, on the morrow, we'll tell.
27441|On the morning of the morrow I, the sea-king, lay in a lonely deep,
27441|And heard a strange old song;
27441|A sound like thunder came from the heavens, and it seemed the old sea
27441|The night has long been cold and dark and dead,
27441|Yet a ghost-like light gleams from the eyes of the stars,
27441|And a strange old song I heard.
27441|The starry heavens grow dim;
27441|The waves are foaming white and red;
27441|But I, in a lonely, weary way,
27441|Hear the great waves roar and roar,
27441|And the old sea-song still sounds on
27441|In the lonely hours of the night.
27441|Sweet-voiced as the misty-colored birds,
27441|The wind on my beach has been singing;
27441|Ah! who is this that sings of the sea?
27441|What are the songs of the sea
27441|That I have heard afar on the shore?
27441|'Tis a ghostly thing the winds are singing:
27441|'Tis a strange old song; O who is this that sings?
27441|It is many a summer o'er the waters
27441|After the sun has sunk,
27441|The sea-bird, high in heaven,
27441|Has never been seen by mortal eye;
27441|But he has been, and it has been well
27441|The waters passed, so he would stay.
27441|With the leaves of the forest all glisten,
27441|And the star of the sky above
27441|Looks lustily with his golden bars,
27441|And the trees are bending low;
27441|He brings the night a blessing, and he makes it clear,
27441|He lulls the weary stars to slumber,
27441|And he lulls the waters under till dawn may keep
27441|The dawning of the day.
27441|

Generating samples...
======== SAMPLE 1 ========
 let you know
9578|What God will never do to you in trouble.
9578|If it's through some brave, stalwart soldier-man
9578|You strive to follow, out-fighting him will spring
9578|Like the break in a wall the water makes
9578|Just when it's nearly over!
9578|A soldier's a soldier for life, and must
9578|Rise up on either hand, firm and growing,
9578|To the work he's called to do, unshaken!
9578|Your country calls you to it; rise up and do
9578|Whatever it asks of you.
9578|It may not be great fighting; may be home drubbing;
9578|May be a life of disgrace and shame;
9578|But never, never, last of any man who strives
9578|To do God's will for country or God's will for home.
9578|Your heart is in it; up and do your work!
9578|God bless you, soldier!
9578|And when the shadows lengthen
9578|And it's very dark inside,
9578|And you hear the droning of traffic in the street,
9578|And the wet fog of the night is about you,
9578|Remember that you are heroes and that your work
9578|Is never done well enough;
9578|That your battles of yore are not lost though you grope
9578|Wildly at your work, nor yet finished nor done;
9578|God bless you, soldier!
9578|Never let your fame be counted
9578|A victory or a defeat!
9578|God loves all great and mighty men,
9578|Not great or mighty in the least,
9578|And He knows that among the sons of men
9578|There are some who cannot reach Him!
9578|"He hath called his seer."
9578|Now that I've tried, God rest my soul!
9578|My feeble prayer failed of a dose.
9578|You'll find a lot of soldier-people, I'm told,
9578|Who'd like to be really great, and they'll try.
9578|I had a vision of a vision once
9578|Of two white boys, one who wore a hood,
9578|And one who, more like a black-hooded ghost,
9578|Held up in his hand the stick of wood!
9578|A soldier boy from the house before me
9578|Stepped forward in his work-shirt white,
9578|And, holding up my old rifle and my old butt-stock,
9578|He turned the wrench of the safety down!
9578|"You'll never," said the seer, as I stood at attention,
9578|"See such a rifle anyhow!"
9578|And that was a vision false and dim,
9578|An apparition of the past,
9578|Of a gray boy from the village of my childhood,
9578|And of my boyhood at home and here,
9578|Who had turned the wrench of the safety to decide
9578|Whether from that dark future or no,
9578|The rifle I would choose at the other's risk.
9578|Not much longer I had written to Colonel Taylor
9578|In the hope of getting prompt reply;
9578|But I found in Colonel Taylor's file a page or two
9578|Where I could search; and there myself said he,
9578|"If a black boy from the village of my childhood
9578|Were to go down into the world to-day,
9578|Would you send up your heart to him? And then
9578|Imagine how the earth would hear and bow,
9578|And smile on you and help you up, and set you free,
9578|And then send up its pitying bosom to you!"
9578|The sun sinks mournful down on the dying of day,
9578|And the leaves are turning into night;
9578|But let not your heart fail
9578|To mourn forever the boy,
9578|Who died before his birthday,
9578|For he dreamed of a great future, too,
9578|And dreamed that the world was his to share.
9578|He dreamed that his step was on the way
9578|To that bright and glorious thing,
9578|Where all who are gifted, gifted, shall go,
9578|To the far, far shore,
9578

Calculating validation loss...
100%|███████████████| 40/40 [00:17<00:00, 2.34it/s]
[658000 | 3737.42] validation loss = 1.93

python src/generate_unconditional_samples.py --top_k 40 --temperature 0.95 --nsamples 50000 \
    --batch_size 10 --model_name 345M-poetry

======================================== SAMPLE 530 ========================================
16265|In their little room with the door ajar
16265|And the candle hanging on the wall ajar,
16265|I have come across the word "Rise"
16265|With a face as grave and flat as you please.
16265|The one thing I remember of "Rise"
16265|Is the way it makes you feel--so bad, so bad.
16265|And I've come across many words to-night
16265|That are so like "Rise"--so like--so vague, so vague.
16265|"Elegance," and "Artistic Vigour,"
16265|But "Rise" is far above the rest,
16265|And I cannot hear--or see--the word,
16265|I will just stop here (I'll stop if I can).
16265|If you don't know what "Rise" means, try.
16265|"Rise" says--but don't ask.
16265|Well, I believe that I do--
16265|And if you don't know what "Rise" means
16265|You won't either!
16265|It's when the sun makes his round
16265|On clouds a-shine;
16265|And the little stars come out
16265|From under the silver dawn
16265|To shine about me so.
16265|When the sun puts out his light,
16265|And closes his earth-door,
16265|And with shadows blackens the blue
16265|Of heaven's blue;
16265|When all the stars I own
16265|Have put their clubs in bed,
16265|And I lie just like a log
16265|In the deep, still earth;
16265|When I don't hear my mother
16265|Any more in the hall;
16265|When the cat is out of the kitchen,
16265|And my brothers and I
16265|Are up to something unsaid
16265|That we've long been keeping--
16265|I have been like to catch a cold
16265|For weeks beyond reach.
16265|And I know that this is so--
16265|And the fever that wrings
16265|My bones is a cunning one
16265|That will not go well
16265|With the morning.
16265|And when I feel all warm and well
16265|And young again,
16265|I have a thought--and I say it--
16265|Of the night that's come.
16265|'Tisn't much: a summer's night--
16265|To-morrow!

======================================== SAMPLE 570 ========================================
18007|With a face of a smile, and a mouth that was redder than wine,
18007|And a hand, and a mouth that was whiter than snow,
18007|And a voice that was softer than dew of the morn!
18007|And I leaned my face between his, and I held him--I
18007|Holded him, and I kissed him, and--ah, there's a stain!
18007|I held him, to prove him faithful, while I dreamed
18007|Of a face of a smile, of a mouth that was redder than wine,
18007|Of a hand, of a mouth that was whiter than snow,
18007|Of a face, of a mouth that was blacker than black.
18007|But all in vain; and the moon, the moon, the moon
18007|Came a-drifting, like a careless sister, and stole
18007|Away from the village, and left them aghast,
18007|When they found him at last in the house of his wife--
18007|Sick with love, with a heart that was blacker than black.
18007|One moment as he wept, and then, like a priest,
18007|He stood weeping, with a pallor on his face,
18007|He stood with his head bowed, and prayed, and was silent;
18007|Then he turned to his wife and said, "Wife," and I
18007|Shuddered--and kissed her, and said a poor thing, and bad;
18007|And--there's a stain!
18007|And then came the moon that came so seldom this year,
18007|And vanished so swiftly that I forgot to say "Good-night."
18007|And she answered, "Kiss me, my sweet; it is time."
18007|And she rose and went to the door; and then--I cried,
18007|And--there's a stain!
18007|Then I thought that I had forgotten all,
18007|All that I had felt and done and said
18007|For I had little to lose, and I had not much to say;
18007|But I knew what did come after I turned in the door,
18007|And that is, that the stains will never go away.
18007|What matters it whether the moon come again,
18007|Or the sun rise, or the little birds sing;
18007|That the little things that I have known and done
18007|Will hurt me still, and stain my lips with black.
18007|The moon is low, and from the windows, low,
18007|The little stars wheel in their canteens,
18007|Ripening for a night when the world is gray,
18007|In the west, far, long ago;
18007|Like flowers, they come, the stars of midnight, bright,
18007|Like flowers, in a cloud of gold and snow.
18007|The stars of midnight hang like ripples of wine;
18007|The flowers have bloomed and vanished away,
18007|Too early to be precious to us all.
18007|But the things I love the best, they have not died,
18007|And shine through the twilight, gold and red,
18007|Too late for our love, too soon for our tears;
18007|So dark and drear for their tender glow,
18007|But yet they are bright, for they will not die.
18007|In the land of shadows I will seek
18007|The star of night that shone above her urn;
18007|And she will laugh in the darkness again,
18007|When I have been gone with a heavy heart,
18007|And she will smile in her lonely home on the shore,
18007|When I shall come no more.
18007|Out of the West, out of the night,
18007|A star has come.
18007|He has rolled in glory before
18007|My soul and I.
18007|I think I have felt him before,
18007|But I never was human.
18007|Now the stars are turning into night,
18007|And the clouds are brightening.
18007|For I never was half so fair
18007|As he now is;
18007|And I never shall see him, only,
18007|Till our bodies meet, heart to heart,
18007|In the bosks of love.
18007|When all is

======================================== SAMPLE 610 ========================================
29700|Hark! from those shadowy depths thy voice
29700|Mournfully echoes, "AUTH".
29700|That was the night when from her fane
29700|Mangu Damoneo, her priest,
29700|Sang by moonlight in the glade
29700|Of her garden; but, alas,
29700|Her lovely form was gone.
29700|The night of sudden twilight,
29700|Of sudden funeral, laid
29700|The flowers of her beauty by,
29700|And left her lying dead.
29700|The mourners, with no pity
29700|For that fair maid, who had died
29700|So young and sweetly, they
29700|Drew from the crowded bier.
29700|Now the long twilight mourns
29700|In dark convulsions slow,
29700|The sun, whose brightness, when it climbs
29700|To meditate its last,
29700|Turns the great globe of heaven
29700|To a vast blackness; and the moon,
29700|In the blue distance lost,
29700|Waves her orb all palely pale
29700|O'er earth and ocean's bed,
29700|While, at her last sad funeral,
29700|The waters are still.
29700|"Where, O, where," the poet cries,--
29700|"Where is she, whom so fair,
29700|So pure a form has laid,
29700|Whose heart was true, and tender,
29700|And soft as summer air!"
29700|Yet there is sorrow in the words;
29700|For in the middle space,
29700|The grave is silent, but the sea
29700|Is louder still than I.
29700|The poets write how fair
29700|She lay in moulded tomb,
29700|A shape of beauty, soft of feature,--
29700|Yet still she's dead!
29700|I look to distant lands,
29700|Yet still I see her face,--
29700|How fair she lies in tomb!
29700|And all that Nature left,
29700|Which never came to pass,
29700|Is on her cold dead breast
29700|That cold dead face seems to look,
29700|As if his own it were.
29700|From sea to sea he sailed,
29700|And then upon the shore,
29700|The man-child found his mother's breast,
29700|The mother's bosom bare.
29700|On many a foreign shore,
29700|When sorrowed he passed along,
29700|The deep-embattled ocean cried,
29700|"A bitter heart, I've had!"
29700|The storms and the tempests of his youth
29700|He turned upon himself alone;
29700|And though the storm and the tempest had
29700|Their rage, he had passed with the wind,
29700|With God,--his only sign;
29700|And, ere the cloud of the tempest blew,
29700|His soul was with the world at play.
29700|He looked to the stars, and the stars smiled,
29700|And the moon in the heaven looked;
29700|And, as he looked, he beheld her light,
29700|And all the heaven smiled with him.
29700|When winds and tempests fly,
29700|When floods and fires fail,
29700|As their wake doth meadow and fen,
29700|'Tis the man-child's heart that craves.
29700|And I--I shall be bound,
29700|With the hoary-headed, strong, old,
29700|To earth, and the graves of the dead,
29700|Whose feet are mowed down, as they lie;
29700|And I shall rest my weary head,
29700|In the silence of Eternity,
29700|In the peaceful arms of God.

======================================== SAMPLE 620 ========================================
38566|and of the two main branches of the lyric poets, he is
38566|of the first. This latter is more like a novel, the former
38566|a mere exercise in poetic technique. It is true that some of the
38566|longer poems are only in two or three of the seven acts,
38566|(as 'Cytherea and Amphithoüs', 'Odysseus and the Daughters of Pose)
38566|The metre of the whole of the tragedies is at most
38566|two or three distinct feet, two or three acts, or perhaps
38566|no more than one line of one act. But there is always no
38566|clear, distinct point of passage, and the metre may change
38566|after the action has begun, and in doing this the danger of
38566|distinction. As regards Lucan, we might as well look to one of
38566|the last days of the Republic, the 'Ante-ratu' of The Deed
38566|(cl. 27)--'Ad locum triplice trieter hoc sibi sibi dicas, etc.
38566|Præestus huic erat precatus in sæpe Lucania per Bettini
38566|Ergo sæpe diadema'--quoting an asp of the same name as the
38566|antagonist,--'Ad mihi quærent horribilem fratres' being a better
38566|means than the one named 'Hæcam.' And thence we can infer from this
38566|that the 'Ante-ratu' of the other tragic poems was a regular and
38566|accidental epigram of Lucan himself,--the kind of thing which
38566|Sextus in his Annales, 65 says of the poets of A.D.
38566|Mihi sæcula, quærent Hæc morbis,
38566|Nec coelo mutum modo vel fratris
38566|Rara ferus.
38566|If it is right to treat of the tragic poets of a later period, it
38566|should not surprise us that Lucan is so honoured in our day. Already
38566|GREEK, and still more after that, in the days of A.D. 60 and A.D.

======================================== SAMPLE 670 ========================================
36773|But not with the first men of Greece;
36773|Not so much the Greek itself
36773|As what this Greek gave and took from him
36773|His power to change what the past had been.
36773|The past he made more perfect than it was;
36773|Not for the beauty of the things,
36773|Not for the new things it is, and strange;
36773|But for an inner sense and power,
36773|The force of a spirit to penetrate
36773|Truly into Nature; the power of Nature's mind.
36773|But these are, in spite of words, a vain,
36773|A futile spectacle!
36773|For what were all
36773|But scenes of dream
36773|And scenes of glass,
36773|Till that which best is​ is at last all?
36773|The image of the true?
36773|Not this, as of old; not this, as of old
36773|When, leaning on his pike the Spartan king
36773|Spoke to his multitude like one man alone.
36773|Nor is it so with me; since naught can prove
36773|My nature's image; all I feel is known
36773|And knows what it is: the image of man;
36773|The body; the sense; the mind; the will; the blood;
36773|And so goes ever on to the dark heart of things.
36773|But this image, as the great natural bulk
36773|Of all the world, and the universal frame
36773|And home of all things, is a wordless sound,
36773|A little, but a strong one, heard and seen
36773|In the unspeakable calm of the supreme;
36773|That, like one man's word of it, "A little more."
36773|The soul shall know
36773|How, like the world of things,
36773|And like the thoughts,
36773|Like these our thoughts, our thoughts are many,
36773|Like the world of things,
36773|And like the sounds,
36773|Sounds of life heard and unseen,
36773|Like the world of sounds,
36773|Sounds of thought and thought, they are the same!
36773|Though what we think is like all things, like we,
36773|Though we be like one thing, not like all.
36773|This is the meaning of life.
36773|For each is like to him, like all;
36773|Though all like him are like to him, not all.
36773|Not like, but like how, and with these two
36773|The earth is like most in the sense of being strong,
36773|The rocks and the sky, sea and shore,
36773|And air and water, and all things like to blood.
36773|_With many a rose for the dead_
36773|_Is laid the flower of mine._
36773|_The flowers, the rose, the sea, the air, the water, the land._
36773|'For we are the pictures of our lives, and our ways
36773|Make up a universe.
36773|'We live life out in the open,
36773|And leave the rest
36773|To fancy._
36773|_The mind-for-life of the sea._
36773|'The mind-for-life of the sea
36773|Takes place for us, for us
36773|The little seas, the little woods,
36773|The little leaves, and the little birds,
36773|The little fishes, the little things that live
36773|The round world over,
36773|And love us with its love of the things we see;
36773|And the mind-for-life of the sea
36773|Is our outward life,
36773|And we would leave it undone, undone,
36773|Right, by ourselves,
36773|For any one to do it for us,
36773|Without any help,
36773|For any one to bear such pain,
36773|That we could never be content,
36773|For any one to stay,
36773|As we have been,
36773|And go on being, and on being.'

======================================== SAMPLE 740 ========================================
19385|"While his auld foot beats thae ances,
19385|Ah! daur ye bee at the auld bar?
19385|Ah! daur ye bee at the auld bar?,
19385|An' if ye have ten min'us ances,
19385|Daur ye be at the auld bar?
19385|There's a cauld, dingy bar,
19385|An' ae blythe young lady
19385|Was a' in his auld house,
19385|He had a fondness for her,
19385|She was his auld ha'."
19385|"I am a king o' the Irish clover,
19385|An' nae sic clogs o' gowlds I ken;
19385|But, whisht! I'm wae to hear men bleedin'
19385|At my reign o' the Irish clover,
19385|An' to see women's looves like wee diz,
19385|When the land that I rule is the land o' clover!
19385|I hae a wife an' a daughter fu' o' age,
19385|She lives down in the clover ha'--
19385|The land that I rule is the land o' clover."
19385|Oh, I was sae blest to leave the play,
19385|That, like a ghaist, rusts awa!
19385|On a windy hill, I heard the snowdrift sweep,
19385|But, oh! how gleyerfu' it was to die!
19385|I set me down upon the heather, where
19385|I heard the wild bee as it swarm;
19385|While the linties all were on me blithe and bonny,
19385|In a bonnie ring I lay.
19385|I slept the sleep that fyfu' the night end,
19385|I thought that I was left alone,
19385|But in a swoon I see the light is glint,
19385|I saw the light was west.
19385|My gowld was yellow, my heart was saft,
19385|My hopes o' long ago were fled--
19385|But lo! ilk bird's sweet singing on,
19385|I heard it far and near.
19385|I think upon my past, the while I'm casting
19385|Ae thought o' Heaven above me, yet
19385|I'm ever glad to be gane!--
19385|The land that I rule is the land o' clover!
19385|'Tis a saft breeze that gies me a kiss--
19385|'Tis a saft, fresh caftinge on my love,
19385|That waves upon my face,
19385|As gladness I do kiss, and he kiss me,
19385|Though he be sae slow.
19385|I canna get used to a kiss from your lips,
19385|But I'm wae to think that I'm blest,
19385|Whar the winds are blowing, and the rain's in bar;
19385|That they may be a-kissin' and a-kissin' at the same.
19385|The gowan-stalks grew high in the heather,
19385|The kye lay snug in their beds;
19385|And ne'er in the days o' our childhood
19385|Were we wint o' such joy;
19385|For, as life and its pleasures might come annee,
19385|So we fain was we to lie where the heather lay.
19385|Then he leaned o'er the heugh, and he laughed a hearty laugh,
19385|And the kye looked aghast, and the deil look'd asker anither,
19385|For their heads were sune to change.
19385|But the gowan-stalks they turn'd in the air,
19385|And the kye wad sieve awa',
19385|For, like a saut bird, O, saut were the tears
19385|That we shed as we fain had we seen, a wild bee on,
19385|But what could be a bee on!
19385|And sair, sae we sieve awa!
19385|And sair, sae we sieve awa!
19385|And we fain would lie where the heather lies, close

======================================== SAMPLE 800 ========================================
27669|We never say "Thank you"
27669|(Except when it's very late!)
27669|But he's the kind that would go
27669|And do his very best,
27669|If you'd give him a hand
27669|If he only were mine.
27669|_And sometimes when we sit around the fire
27669|We talk of a thing or two:
27669|Of the boy that was mine when I was young--
27669|(No matter!)--we talk of the boy we used to know,
27669|When he played in the wood,
27669|When he went a bowing round,
27669|In the golden olden time;
27669|He was just another boy._
27669|It's a fine old wood and I've tried to walk
27669|Through the boughs above a few leaves,
27669|And I've looked where there's little green places,
27669|To the shade of a great blue sky.
27669|There's the sea, there's the meadows, the white road and yellow sheep,
27669|And there's the old, old, old road where I met him last._
27669|'Twas a wood that he liked; it seemed just green and fair;
27669|And he liked the boughs above the great blue sky.
27669|'Twas a field we used to pass, now he is far away;
27669|And he loves the meadows, the white road and yellow sheep.
27669|And he loves the field when he is dead, but he'd rather be
27669|With his little boy there in the woodland far away.
27669|When he's gone
27669|He's a rich young boy sitting in his mother's lap
27669|And that is why he'll not forget
27669|To kiss and to hug me when he's gone._
27669|_Oh! would you'd come back, this would-be friend,
27669|And we'd talk of a time when it hadn't been too hot
27669|Since we all set out; and how we'd walk
27669|Along the road again, where he'd left us.
27669|And when we'd reached him in the valley he'd say,
27669|With a smile, "Good-by, my dear, good-by!"
27669|And look at me with his dimpled smile;
27669|And I think we'd talk and joke about such things.
27669|And maybe we'd play at our old home game,
27669|If you'd come back
27669|And help to share in our old home joy.
27669|And when you are back
27669|It's just that we two can walk,
27669|Along the road again:
27669|We can watch the sheep and the grasses grow,
27669|And tell each other stories and listen to the wind
27669|Crying to the woods and calling in the trees
27669|Through the blue day, "Old dear, old dear!"_
27669|_As we were riding down that lane of blue,
27669|O'er the brown hill's crest;
27669|And there in the valley between our feet
27669|There stood a house._
27669|'Twas a tiny, simple-minded place,
27669|With a fence, and railings, and railings more,
27669|That made us feel so small.
27669|It could only be called 'Half Price' Town.
27669|There were houses up and down,
27669|In the distance we could scarcely see,
27669|To the horizon's end.
27669|_We sat, in the shade of a timber tree;
27669|Our eyes half closed, and our faces bent;
27669|And with a heavy heart we thought of home,
27669|And of the woman who'd died asleep,
27669|In the house we had grown to cherish.
27669|And the voice was quiet again;
27669|In the silence of the night,
27669|In the light of the moon and stars,
27669|And the night-wind we could feel
27669|The pulses of the sleeping and dying,
27669|In the quiet, moonless house.

======================================== SAMPLE 950 ========================================
1279|The King of Wales is a very clever fellow;
1279|He's sly as a cat, and a thief as well,
1279|He's like to set round a lamb a score,
1279|To take and turn round, and a-mimicking her tone.
1279|But to crown my Lord of Leven's min' fun,
1279|He's just like a dunce in w'at he can do;
1279|He's all in a twirl if he's going to be tail,
1279|For the man that heaps things at Lord Brougham's feet.
1279|Lord of the Whinnybone and the Hooplebooroo!
1279|You're a wit all as fine as Sir Thomas Suckly,
1279|And I'll venture to say, in my noddle,
1279|That when all the rest are out to piss,
1279|Then your Lord of Leven is just the man to go.
1279|Now, the best and the brightest of our ancestors,
1279|Who bore the race formerly known as Bow-wow-wow,
1279|Were all of one gender, and of one sex,
1279|And yet they were men of many other breeds,
1279|And more and more there was variation in their breed.
1279|For example, of the male, who was black and of white,
1279|There was also a fair breed of females all over the country;
1279|And yet 'tis a fact that the best men of the day
1279|Were all of one sex and of one species,
1279|And yet they were men of many hundreds thro many centuries.
1279|So there never was anything like it--there never was,
1279|In any age or clime, a breed like to the first:
1279|All were of one sex and of one sex combination--
1279|So, you see, they were men of many hundreds,
1279|That were all of one sex, and of many thousands,
1279|Whose species combination was myriad times more numerous than yours.
1279|In every locality from Cornwall to the Cape,
1279|There were divers sorts of males, and divers sorts of females,
1279|In every locality and every rank and class;
1279|For example, of the first five generations,
1279|There were divers sorts of weds, and divers sorts of brides;
1279|In every locality, and every rank and class,
1279|There were divers sorts of housings, and divers sorts of shoes.
1279|And therefore in every locality and rank and class,
1279|There was variation, combination, and change of occupation:
1279|There were divers sorts of women, and divers sorts of men,
1279|In every county and every town and town-house too.
1279|The first five generations were all of one sex;
1279|And this was probably why before the time of the fifth,
1279|There were divers sorts of brides, and divers sorts of men;
1279|For example, in the first five generations,
1279|The wife was the opposite of the husband--most probably;
1279|And the husband was usually a bachelor, or a man,
1279|With another wife, and the same sex as his wife,
1279|In every county and every town and town-house too.
1279|In every county and every town and town-house too,
1279|There were divers variations of dress and apparel;
1279|Some were dressed in crape, and crape-cloths and bonnets,
1279|Some in kirtles, and some in hose and hose-at-heel;
1279|Some in high-heels, and some low-heels, and some hose-nigh-loose,
1279|Some in high-heels, and some in hose-nigh-loose;
1279|And some wore noddles trimmed with blue or white,
1279|And some with knobs, and some with rings, and some with rings;
1279|And therefore there was plenty of noddles for all ranks and classes.
1279|The last five generations (which I shall say was one
1279|generation before the last), were all somewhat further off
1279|than the last, and thus left me to interpret
1279|The names of the divers divers divers generations.
1279|A woman in red and white was as happy as a girl;
1279|

======================================== SAMPLE 1060 ========================================
5185|"To the fish-lake of Pohyola,
5185|To the lake of honeyed Hiisi;
5185|Weep Iwi, weep upon Lempo,
5185|On the blue rock of Hiisi;
5185|Weep the god whose tears enrich;
5185|Weep the hero who subdued
5185|By his iron-handed arms
5185|This enfolding water-brook,
5185|By his magic hurl the rock
5185|To the upper deeps of Hiisi!"
5185|On the floor of clay they throw it,
5185|On the stone on which they write it,
5185|Spake these words in magic measures:
5185|"Lo-ye-Yo-Ru-Raka, son of Hiawatha,
5185|Take this magic measure,
5185|Words of ancient prowess!
5185|Put your senses into it,
5185|Take these mental forms into it,
5185|Turn to toe the enchanted drink
5185|Of the ancient, good, and wise Wabun!"
5185|As they twirl the mental measure,
5185|As they pour the mental formulae,
5185|Words of magic potency
5185|From the wizard's magic red beard
5185|Stream into the magic waters,
5185|Rainbow-colorful draughts of rum
5185|From the six-pending arts of brewing.
5185|Thus at midnight, hour of darkness,
5185|From the six-pending arts of brewing,
5185|Munchausen' of the spirit SAMURAI,
5185|Bitter his bitter sorrowful experiences,
5185|Tears flow to see the brewing measures,
5185|Casting him dark as night to HIAWASH,
5185|To the dancing-bower of Winansi,
5185|To the palace of the Song-element,
5185|To the home of ancient Wabun.
5185|There he sees the wizard fire-breather,
5185|There he eats the fruit of fire-fruit,
5185|Warm the water in the blue-smoke,
5185|Honey of the fire-flies, Sahwa;
5185|Nevermore to rise from Sahri
5185|To the glittering skies of Hiawatha,
5185|To the shining islands in the ocean,
5185|To his home in Winansi's honey-lands.
5185|True he sings no more of former loves,
5185|Sings no more of former partnerships;
5185|Beauty and truth the singer forsakes,
5185|Joys and dreams of CHEWBEL tree-top hollow,
5185|Thus again he sings to little honey-paws,
5185|Sings but one sweet thing, one only,
5185|Sings of the honey-pastoral flowers,
5185|Sings not of the meadows golden-rod,
5185|Nor the soft, corn-fields resting in slumbers.
5185|Young again the wizard sings of combing
5185|Sunny downs and forests of awaking,
5185|Of the golden balls in linden-groves,
5185|In the nooks of meadow-lands adorned
5185|With the purple blossoms of the ash,
5185|And the golden globes of chrysanthemum.
5185|As he sings, he flays and bleaches piarmate
5185|Peppers, onions, garlic, and ears of corn,
5185|For a meal to make his night's banquet.
5185|Straightway Hiawatha asks the artist,
5185|Places in his basket heaps of grain,
5185|Hangs himself to earth in alley lowly,
5185|Lays his hands and knees upon the rafters,
5185|Drops his baskets of grain into annihilation,
5185|Sings in low, third parts, low alto-dingule.
5185|Finally he hastens to the stable,
5185|Finds the black-frost of the winter gathering,
5185|In the center of high rafters framing;
5185|In the center a famine-gathered mass,
5185|In the center a man and his banditti,
5185|In the rafters great store of grain reaped from snow-sledge.
5185|Homeward goes the artist, homeward.

======================================== SAMPLE 1420 ========================================
1381|The loon of the moon!
1381|In the night, to me,
1381|The air was full of stars;
1381|The wild bird piped upon the branch,
1381|And the blackbird, tuned his song
1381|To the sweetest note
1381|That ever a lark knew!
1381|And I heard it ring, above the sea,
1381|In the land of the dreaming sea,
1381|In the land of the dreaming stars!
1381|A star is born when the summer moon
1381|Is full in the sky:  she wets her tears
1381|To kiss the star and sigh to hear
1381|One sigh from a thousand stars.
1381|A star is born when a nightingale
1381|Sings in a blue-bell tree; and she
1381|With such a song and such a tune
1381|Fills all the fragrant heart of the wood
1381|With such a fragrance as is there,
1381|And she sings in the heart of it.
1381|A star is born when on the wildest seas
1381|An Indian maiden dances alone,
1381|Pale as a ghost in the moon's eclipse,
1381|And the stars and the wind sing from the foam;
1381|And as the moon is dimmed in the heaving deeps,
1381|The heart of the maiden lies high and free,
1381|Shaking her curls in the beating breeze,
1381|And her eyes are bright with ecstasy.
1381|A star is born when the wind-flower sings,
1381|Or the hawthorn bathes in dew;
1381|And the nightingale sings like a bird in the sky,
1381|And all the stars and the nightingale ring,
1381|And the heart of the wind-flower sings.
1381|A star is born when the snow-white swan
1381|Drops her white soul to the river:
1381|And the winds and the water sing with glee,
1381|And the white swan lifts her face to the sun,
1381|And the stars sing out in the glory of day:
1381|And the heart of the swan is happy in heaven,
1381|And the heart of the river sings in the sun!

======================================== SAMPLE 1550 ========================================
937|And, as you will, I'll be bound to you,
937|And always be and never be forgot.
937|We were a little band of heroes then,
937|With our flags all waving by our side,
937|And our hearts all set on some deeds of fame,
937|And our eyes all full of good-will light,
937|When the Kaiser came along to help them.
937|He was a mighty thing of might,
937|And when with us our flag all flowed,
937|When all men called each other comrade,
937|When the earth was made for happiness,
937|When freedom's dawn was dawning on all lands,
937|Our lives were all to some great cause done
937|We might not be wholly silent there,
937|For our hearts all yearned to join with 'em,
937|And we could not all go quietly by
937|But stood to fight the battle hard to the end;
937|And so, long as we had hearts to move,
937|The Kaiser, to keep back our strong men,
937|Was always there to tell us that we were called
937|To fight for liberty---and all were we:
937|And every man must be a Kaiser
937|When our glorious flag we salute.
937|And now that there's a Kaiser in town,
937|The hearts we had at first to thank
937|For the end of the old war we went on --
937|To keep back the strong men to stand by us;
937|We are so proud to hold in scorn
937|The Kaiser's warning words back.
937|So when you hear our cheering chorus,
937|Look down in the faces of your foes.
937|And when you're called to your father's home,
937|Oh, just smile---and let him go by:
937|It was the Kaiser's own call
937|That your country was called to give.

Tao Te Ching

The Tao Te Ching (TTC) is a famously enigmatic text, written in a difficult style in an more difficult language, and because of the challenge, has attracted many highly-varied translations. (For a 2024 attempt using ChatGPT, see “a wandering mind”.)

Hiræth Weltschmerz compiled ~270 translations of the first verse of the TTC in one text (108Kb/19k words), and compiled per-chapter translations of the rest of the TTC as well (4.8M/870k words). Using TTC snippets to prompt GPT-2-345M didn’t produce good results, so he asked me to train a GPT-2-345M on the TTC to see what it did.

I converted the first corpus to Unix text format, replaced various escaped character entities with their Unicode equivalents, and replaced double newlines with single lines, and trained the original GPT-2-345M (I was unsure if using one of the poetry models would help) with the usual settings for ~6 GPU-hours, at which point it had reached a loss of ~1.8 and I began to worry about overfitting & stopped.

I generated the usual ~1k unconditional samples:

• 2019-07-19-taotehching-ch1-1ksamples.txt

Some training samples:

...
[12999 | 10468.36] loss=2.03 avg=1.96
Saving checkpoint/taotehching/model-13000
Generating samples...
======== SAMPLE 1 ========
 can find a way to rid yourself of these issues.
Forced to see them (the spiritual essence), there we receive freedom.
Free from spirituality, we receive nothing but desire.
These are not separate things. They are one in common. Together they are the Mystery.
The Mystery of the World is all illusion.
Everything we see is but a small part of the Mystery.
The true mystery is beyond understanding.
The mere awareness of the world is a mere shell.
By being aware we become a turd.
Those who try to hide the true inner essence will become trapped in the shell.
Those who hide their inner essences are free to become free.
These two pairs of differentiations arise from the same source: nothingness and everythingness.
Their differences are mysterious, but still they are infinitely subtle.
Beyond spirituality, they are the portal to all wonder.
The words we use to express the nature of these differences are not the same as the words we use to express the nature of other differences.
The same name may be applied to different things.
As a mere passing fable, it is the boundary of mysteries.
The real is infinite, but it has limits. The empty of the empty is the gate to all spirituality.
The name that we use to explain it, is not the name of the real.
It is the Mystery of all things; as "the ground of the profundities of the heavens and the abode of the ten thousand names.
The profound subtleties are discovered from the deepest minutiae.
The Uncomprehending is the gate to all mystery.
The unsearchable is the door of all knowledge.

The Tao that can be spoken is not the eternal Tao. The name that can be named is not the eternal name.
The nameless is the beginning of heaven and earth; it is the mother of all non-being.
The named is the mother of all things. From the named it is known what is above and what is under.
Words are ways of naming; words are names for things.
There is nothing beyond the limits of the mind and thoughts; there is something beyond the limits of the mind.
This vision that can be voiced in words is not eternal vision; this that can be voiced in words is not eternal words.
The indescribable is the essence of heaven and earth; it is the Mother of all things.
Exclamatory to the absolute is the threshold of all comprehension.
Descriptive to the absolute is the gateway to all differences.
Non-being is the limit of what is known. Exclamatory to the absolute is the threshold of all transcendence.
Non-being is the origin of heaven and earth. Exclamatory to the absolute is the gate to all diversity.
The name of the universe is not the true name of the nameless.
The Nameless is the gate to all mysteries.
Having a body we see, The body is the door of all wonder.
Hence, The unmanifest is the gateway to every wonder.
Names are ways of naming; Names are limits of names.
Unmanifest is the gate to all things.
The unmanifest is the mother of all things.
Nameless it is the mystery of the universe.
Existence is the domain of all that is.
Obscure is the domain of all that is known.
"It" is the beginning of heaven and earth.
The manifest is the Mother of all things.
This unity, Is the beginning of the universe.
Non-being is the Mother of all things. Non-being is the Mother of all things.
It is the mystery of the universe.
All things are (A Real) if they are perceived as a number, or unity, The mystery is the key to all things.
Names that can be names are not real. Names that cannot be names are real. Nothingness and all its aspects can be perceived as the subtlety of the mind.
The Namelessness or totality is the essence of all things.
Names that can be names are not real. Names that can not be names are real.
Nothingness is the Mother of all Things.
Nothingness is the mystery of all things.
Non-being is the Mother of all things. Non-being is the secret of the universe.
The Teaching which we can name is not the Mystery we can observe. The name that we can give is not the name we can give; but the Name that we can call is the Name that we can see.
The way that we can see is not the Way of the Past; it is the Way of the Past that is manifest, But without lasting, though enduring, we shall see the Unseen.
This we call the Mind, The Mother of the Past and of all Past: It is the gate

Calculating validation loss...
100%|███████████████████████| 40/40 [00:16<00:00, 2.45it/s]
[13000 | 10503.37] validation loss = 1.81
[13000 | 10504.22] loss=2.17 avg=1.96
...

...
[15099 | 12666.34] loss=1.95 avg=1.78
Generating samples...
======== SAMPLE 1 ========


The maiden-maiden is the Mother of the Serpent,
The mother of all things, is called Mother,
Who has been named and is called mother, has no share.
He who has no name conceives
He who has an idea conceives
These two, existing without separation,
Have their root in the same source; In the same sort of source,
The One and the Being of the same and both differing in name,
Reaching out from the same source, Dissolves both.
Through these two there arises within all things a measure.
From these two measure and measure, They are called the World-Dao.
So, just as Nameless, let us look upon the mysterious depths of existence;
As Manifest, the sight of its manifestations.
To rid ourselves of desire, let us view its nameless outer form;
As we perceive its unmanifest core, we see its pure Inner Light.
This is the mystery hidden in the entire mystery.
As for the world, Nameless we see its beginning.
As Manifest we see its progression.
That which we are yet not filled with, Is filled with fire.
The fire of all-pervading Truth, Wherece from without can spring, From which flow all those things that are known.
Truly alone, viewing the Unconstructed, I see a mystery deep within the Unconstructed.
That which is Unconstructed is but the beginning of great things;
That which is Named is but the gateway to great things.
It is by working that the Heavenly Way may become manifest,
That through an effort that We may see its full extent; That is, We seek to see the hidden parts, We may see its mysteries,
that is, We must see the Unconstructed.
In that which is unobstructed, the dawn of all things is brought.
In that which is filled with things, the darkness of all things.
These two emerge and separate, Yet together, All things are called darkness and light.
They are both of One Origin, Which is the Gate to all Profoundness.

The Nameless is the door to Mystery,
The Manifest is to a Reasoned Way the gate to all hidden Mystery.

It is from mysteries which we shall not know, That we may approach to the Eternal, In order that we may obtain the Path of Life.

It is not always the case that, While nothing is experienced, The Essential Principle itself is hidden.
This state of Nothingness is the gateway to the Manifestation of all that is Eternal.
It is the origin of that which is lasting, Therefore it is the Emptiness which is the Gateway to the Finality.
That which is without beginning, It is the gate to all the subtleties of the Eternal;
That which has an origin, Being a process, Is the source of all the subtleties of the Eternal.
If you would know the Divine mystery, Become a Deserter. Life is a spring from a Deeper Mystery.


The unvarying Manifest is the Source of the Unvarying;
That which has a name is the mode of manifestation;
That which is unnamed is the mode of existence.

Unity is source of Manifest, Non-manifest the mother of the manifold.
Thus to watch, to wonder; to have desires and desires; All things, Manifest and non-manifest, To return whence to seek its way.

Tribulation is its mother.
The mystery of the Tao is the Body of the Tao.
The Essence of Tao is Tao, Tao the Cosmic Universe;
All non-being Is its mother.
Non-being is its mother Because it has the Secret Principle hidden,
Though without Secret Principle It is the Mother of all.
That Which Is Deeper Goes Before Deeper,
And That Which Is Secret Goes Before Secret;
They Are Called the One and the Mother of All Things.
The Name Is from Mystery,The Creature from Form,The Origins of all things
To perceive the Secret, Become a deserter
Desire Returns from the Secret: It is its Mother The Secret of all existence.
These words:
"Empty Words"
To see them Is to see the Secret,
To understand The Secret Is to understand The Origin
Its Dynam is Manifestness (Of all)
That is Its Mother (Of all)
That which is empty
Remain a Mystery
Beyond Mystery
Beyond Proserely
With all mystery in Its Depth
Has the Serpent walked
To understand Its Secret
Has become a Serpent
As for that which has a name It is the beginning of All Things
The One without a name has originated All things.

What arose is called the Essence of All Things.
These two that were separate
With opposite names But Ever Dissolving Is the

[15100 | 12682.90] loss=1.69 avg=1.78
...

I then began training it on the full TTC corpus, which was split into per-chapter files. Remembering the problems with run-on poetry, I added the <|endoftext|> markers to the end of each file. (It would be better to add that to the end of each translation, but HW didn’t include any delimiters for each translation, and doing so manually would be too much work.)

This required substantially more training time.

Some training samples:

...
[88999 | 6150.00] loss=2.33 avg=2.44
Saving checkpoint/taotehching/model-89000
Generating samples...
======== SAMPLE 1 ========
 Tao Tao's Way is universal. It covers all lands. Therefore those who realize it can accomplish all under Heaven. Therefore he who maintains the Tao, can be believed to know the Universe.

There is an ocean and a river, and all beneath the heavens aspire to be carried by them. The people of the Tao country follow these rivers and streams, as they may be carried. The people of the man of the Tao country follow the rivers and streams, as they may be guided; and the people of the man of the Tao country follow the rivers and streams, as they may be measured. All under heaven strive to be carried by the Tao Way, as they might be carried by Heaven. The great man, when he knows the Tao Way, can achieve all under Heaven. When a man who has completed the Tao is believed to know the Tao, this is shown by his ability to govern.

All Tao is infinite; to people like it, it is the very mother of all things. That which flows with the Tao is universal; that which opposes it is not. The Great King possesses the Tao Way, so that he can hold everything under heaven. To know Tao, one must be a man of Tao. To know the Great King is the same as to know Heaven. The Great King who understands Tao can hold everything under heaven. A man of Tao must know everything under Heaven, just like he knows Heaven.

True and Eternal Tao Tao can take care of all things that exist, without needing a reason. The person who is able to hear Tao is capable of capturing everything under heaven. Having enough listening ability, the one who can take care of Tao Is able to know all things in the universe. Having enough being able to listen to Tao, The one able to know everything under heaven is able to govern all under heaven. Thus he who is able to control Tao is the one who will be able to control the universe. Whoever knows Tao can grasp everything under heaven. The person of Tao has no limitations, therefore he is able to grasp everything under heaven.

By Tao I can all things be fulfilled, Without needing to get into details. All things call to the Tao Way Without needing to tell the other Tao people to be fit. I know the Tao Way And all things flow with me. If you're also able to think, You can capture the whole Tao Way. Knowing all things That come to you Can help you to know all things that come to others. When one can know Tao You can also master Tao. When the Master knows Tao He can master the Tao Way. And if you are able to master the Tao You can also take all things into your own hands. You don't have to control Tao or all things.

Under Tao, Everything is fulfilled. Tao never requires a reason for doing something. All things--- even beasts and plants--- seek out Tao. But because of Tao's inner being, All things have this one thing to do with it. Knowing the Tao gives you the inner clarity of Tao. But to have the inner clarity you also have to know that you are a human being. So one who is a Taoist learns that he is a human being. The Master of Tao learned that he was a human being. The Taoist who understands the Tao is not burdened by it; He has the inner clarity he needs.

The Way of Tao: All things call to it, Through but one source. Everything that desires the Tao Must either reject or destroy itself. The Great Thinker has Tao, And can guide all under Heaven. To have the Tao Knowledge is to know the mysterious Source. Knowing the mystery reveals the subtle power of Tao. The Supreme Man can conquer all under Heaven. If he knows Tao, he can be truly the Master of All. To know Tao is to understand the mysterious nature of Heaven. If you know Tao, you also can serve Heaven. The Great Thinker has Tao to serve Heaven, And the Taoist who has Tao to serve Heaven is equal to all things.

All things around a "Way of Tao" "says to its people, "everything"s the natural way,"' and everything that should reject it's it alone. Knowing everything about Tao, people accept Tao, and everything under heaven has Tao in it. To know Tao is to know all things. To know Tao is to know that Tao itself is all things, and the Master of Tao can know all things under heaven. Therefore, he who understands Tao can serve Tao.

Tao was born of the Tao Way It's there for you But not through words, and it's everywhere All things call to it, But not through its Tao Words will not accomplish the feat. You cannot hold on to Tao's place without grasping the primeval essence of this force of nature. Know Tao, or your Tao will know the universe It's how you grasp the primeval essence that matters Don't hold Tao hostage and force it to shape

Calculating validation loss...
100%|██████████████████████| 40/40 [00:16<00:00, 2.36it/s]
[89000 | 6186.11] validation loss = 2.44
[89000 | 6186.98] loss=2.34 avg=2.44
...

...
[91999 | 8892.62] loss=2.22 avg=2.42
Saving checkpoint/taotehching/model-92000
Generating samples...
======== SAMPLE 1 ========
 eyes, nothingness. Therefore, he who observes the Dao conceives at his door a room, the ground a mat of grasses, the great thing a hole. He finds no traces of the dwelling of the Sage. He finds in the dwelling of the Sage the Mother, the mother of all. The Sage Mother produces the mother of all. Thus, the Sage governs all things without using arms. He governs them without using the tools of the world. It is he who is the mother of all, and therefore of all things.

To know the Tao is the most important thing in the world. The minute, mysterious details of Tao are what people value. To know the Tao and have a good memory is more important than learning the world. For when the world was simple, The small details were as important as the great details. Nowadays, we are taught to study all things well, but then forget to know what the Tao is. We learn the Tao and love to be mysterious, but when we don't know what the Tao is, we can't be specific. For when we know what the Tao is, it's as if we have a place where all things are and born from. It is the way of Tao.

"The Tao" (truth) is the greatest attribute on the whole world; The minute details of the Tao are important. One who knows Tao enjoys quietude, while one who does not know Tao is restrained. One who is called "taoist" desires the dark side and the subtle, while one who is not called "taoist" wishes to be subtle. They are not the same. Therefore, those who call themselves "Taoists" are not the same as those who are not "Taoists." They who live the way of the Tao do not have weapons, they do not wage war and use the tools of men. In the matter of life, they are different. Thus they are different. That is why they live in the world.

The greater part of humanity cherishes and follows the Tao in common with Tao, even the great and glorious antiquity. This is the reason why people respect the Tao and live in the Tao. Tao is the essence of mankind, or rather the essence of all. It is not the essence of the universe. Only the Son (the Existence) knows the nature of the universe, and the former also what the latter exists. As the Son knows the nature of the universe, the latter also is known. Since that is the reason why the Son has only one nature, and that the latter the other, therefore the Son is distinct from the former in both respect. Therefore, the Son rules the universe and the universe rules the Son. The universe is understood of, by, and for the Son, and he serves the universe with his own hand.

The Tao (the Eternal) is the mother of all Things. Only those who grasp and truly live the Tao can be truly happy. Those who ignore the Tao (the Eternal) turn into chaos. Those who hold fast to belief in false beliefs deprive all beings of happiness. Those who glorify existing things end up in misery. The Tao's path is tranquil; Its ways are pure. It can act with the Tao without deceit. It can behave with the Tao without weapons. It can act with the Tao without instruments. This is what it is like.

To know the Tao is to be the mother of all things, As minute as the minute details, True as being whole. Those who know Tao love not to be ignorant; Those who do not love Tao do not know it; Hence, they cannot be good. Those who keep faith in the Tao get the mystical, Those who no longer believe in the Tao Are called demons. Those who trust the Tao Are bound to bless them, Those who love the Tao are good, Because the Tao is good. The Tao is the mother of all, Which is in all but name.

To know the Tao is to have the mother of all things. Nowadays, the tiny details are more important than the great details. Nowadays, we are surrounded by thoughts and desires. Those who do not live on the Tao have no desires. Those who are good with the Tao see everything in a whole. This is why the Tao is the mother of all things. Because it lives for all things, it does not become a weapon. Because it makes people good with the Tao, they are not evil. Because the Tao is good, it always acts as though it were able to act on its own. Thus it is all-pervading. Everyone lives like this because it always acts according to its own nature.

"To know the Dao is to be the mother of all things. Simple and dark, The depths of Tao resemble life. Who can have true purity without knowing Tao? Those who live in the Dao gain the child, Those who do not live in the Dao suffer in the nurture.

Calculating validation loss...
100%|████████████████████████| 40/40 [00:16<00:00, 2.35it/s]
[92000 | 8928.83] validation loss = 2.44
[92000 | 8929.71] loss=3.07 avg=2.43
...

Hiræth Weltschmerz was able to improve the TTC training dataset by providing the first 41 chapters with the original newlines/linebreaks, and separate translations separated by a blank line, so I replaced blank lines with <|endoftext|> and trained that for ~24 GPU-hours to a final loss of ~2.10. The results read much more poetically, I felt.

For this final TTC-with-linebreaks, I uploaded the model & generated 1000 random samples as usual, but I also generated per-chapter samples. For per-chapter samples, I used csplit to split each file/chapter into separate verses/translations, selected a random 10 of them, and then generated 10 random completions of each one (100 total per chapter):

i=1
for CHAPTER in `ls taotetotalityrevisedpw/*revised.txt`; do
    csplit --prefix xx --suppress-matched $CHAPTER '/^$/' '{*}'
    for X in `ls xx* | shuf | head -10`; do
        echo $i $X
         cat $X | tee /dev/stderr | nice python src/conditional_samples.py \
           --top_p 0.9 --model_name taotehching --nsamples 10 --batch_size 10 \
           | tee /dev/stderr >> ttc-chapter-$i.txt
    done
    rm xx*
    i=$(($i+1))
done

The idea there is that one can write one’s own Tao of GPT-2, going chapter by chapter: select some of the chapter 1 prompted conditional sample completions to create a new chapter 1, and so on, in a way which would be difficult to do with just random unconditional samples.

Downloads:

• model

• 1k random samples (4MB text)

• 100×81 chapter samples (tarball, 17MB text)

Loss: 2.6

Partway through, having reached a loss of ~2.6 (down ~0.5 from the Colab model), we experimented with training our model on a P100 GPU, halving the context window to make it fit, to informally compare its training speed with the swarm. The P100 made little training progress, but it did generate some fun poetry samples (we had disabled the training sample generation for the swarm because generating samples is so slow).

The samples strike me as good, perhaps even better than GPT-2-117M, despite the loss being much worse (2.6 rather than 1.6). Why might that be?

I hypothesize it reflects a weakness of the likelihood loss in terms of perceptual quality: humans are more sensitive to long-range correlations and text degenerating into gibberish than we are to local details like exact use of particles or to slightly better modeling of spelling (which is why stylometrics works). The original OA GPT-2-1.5b achieves much better modeling of long-range correlations and producing coherent text than the GPT-2-117M did, of course. What happens when they are both trained on a poetry dataset? It is the tale of the tortoise & the hare, or the bias-variance tradeoff: the GPT-2-117M is weak, bad at long-range modeling because of its small parameter count & shallow layers, but the benefit is that it can learn quickly about local details like spelling, and, achieving good prediction there, converge to that 1.6 loss; GPT-2-1.5b starts off good at long-range modeling and good at short-range modeling, and must tradeoff learning both from its limited training, thereby achieving mediocre performance on local correlations and thus mediocre loss, even though humans reading it are impressed by the thematic consistency and relative lack of ‘gibberish’ (locally but not globally consistent text).

An additional issue here is that the GPT-2 models are not fully trained: as the GPT-2 paper notes, “All models still underfit WebText and held-out perplexity has as of yet improved given more training time.” (The difficult of training such powerful LMs to convergence was also noted by the MegatronLM researchers, whose MegatronLM-8.3b model was still learning rapidly when they ended the training—despite use of NVIDIA’s DGX SuperPOD with 512 GPUs.) So some of the finetuning here may also be finishing the GPT-2 training.

I selected the following from its training log.

A commentary on Oda Nobunaga’s mysterious betrayal by a loyal general?

"Oda on the Pavement", by Mark Bibbins [Activities, Jobs & Working, Social Commentaries]
The first time I saw this sign someone's blood
streaming down the pavement, I didn't know what to do.
I walked as quickly as I could toward the body
and almost didn't see the second because it was going by
so quickly I had to blink and it was gone.
But this is what I saw when I got there, a corpse
laid out flat on its back, its legs
bent at the knee, with a blue tattered blanket
crammed in the small of its back.
I said to my wife sitting next to me, don't you
think this is awful? She said, Mark, it looks like
just a trick he's done.
I said, it looks awful real to me. The man has
his back turned to us, like he is not even thinking.
As I am, I said to my wife, is this what you have
been waiting for? Is this what you have been waiting for?
The man was lying there not even looking up at us.

In the Golden Age, when the people of the Yellow Valley were instructed by the sages of antiquity:

...That the earth would then not open
(O, but to hear our fathers speak!),
The angel of the waters spake.
For in that time the race of man
Sought the groves, each his own dwelling,
Where were then fruits to satisfy,
Where were trees in plenty springing
To furnish them with fuel,
And so the forest grew,
And o'er it all was running
The river that we see.

Then did those ancient sages
Order their days as was fitting;
And whatsoe'er they looked
On was ordered rightly; all
Taught each what was fit for living
And working for self;
And now we have come to the end,
But this is what their teaching
Said for our instruction,--
That we know and do not know,--
And yet we know and do not know!

When we have found that which is right
The will to act and not to act
To be the chief and not the end,
The day and not the hour,
The joy and not the pain,
When shall we find this, then at last?
Why then at last at last?
There can be no yesterday
If such be the age of wisdom;

There can be no time or place
If such be life and work for man;
There can be no man or man's child
In which he has no friend or friend's child;
Nor yet a man or man's son,
Who to his work does not go.
Thus the sage's order is
And what the sage does--so he does.

Romance?

...I know my father taught me,
To count your blessings and forgive
The people who are wrong;
But I think, when my day is done
And life is all in strife,
Perhaps the one good man who isn't yet
Will find the way to see
Why men die, the reason lies
In the people, never the least in them.

To-morrow it may have been a woman's hand
Caught his in the doorway,
And I may laugh to-night at a stranger's sigh,
Just as I did yesterday.
But what if he and I are dead?
What matters it?  For God's sake, listen to me!
If God finds out
That I am not as I should be,
Then I shall surely pay the price--
That is if He will be just.

I think my father loved me,
And was glad I was his son.
I say to-night I see it every day.
I say to-night I know it all.
And why should I fear
What my son thinks, what he does?
I should be mad to doubt it,
For aught my father says.

I shall find out when I grow old,
And say to my soul, "Be glad
That things that are may be,
For if things were worse, why, we'd have things worse."
And, knowing what my soul knows,
I shall be glad for the things I shall miss--
For God's sake, I shall be glad!

A man's true worth is revealed to you
When, by the light of your own soul's light,
And God's, you look at his;
You can feel them both shine through you
Like a glory you could shatter
Just as a crown could shatter you
Into thousand pieces that were bright
And perfect, and eternal.

And so no other can you see and know
Except his soul.  As the rose of him
Sends back in glory back its hue,
So does the soul of one reveal
The beauty of another.
Be mine the glory to know it best
When my soul shall look at its own.

O great and blessed Lord,
Who made the earth, the heaven, and the sea.
I who am standing in these holy places,
Be grateful

A nice descriptive piece:

...The sun sets and rises in its glory
And sets in its glory only.
And the stars are all that can be seen up there,
That set and rise and come to the earth in their glory.
And the moon, too, sets and rises only,
And comes to the earth in its glory.

The sun sets and rises in its glory,
And when it sets, the night is black,
And when it sets, the day is dark.
The sky is all up against us and over us,
With its starry light,
And all the earth beneath us and above us,
Under the shining sky,
Is black with all the stars.

An elegy:

..."To the Youth", by Susan Greenfield [Death, Growing Old, Sorrow & Grieving, Relationships, Family & Ancestors]
For the youth, who, long ago,
Came up the long and winding way
Beneath my father's roof, in sorrow---
Sorrow that I would not bless
With his very tears. Oh,
My son the sorrowing,
Sorrow's child. God keep thy head,
Where it is dim with age,
Gentle in her death!

And though the years should bear him low
Into the world's dark embrace,
He shall feel more fondly there
Than he did in my bosom then.
Even now that heart, so wild,
Is as a child's, sweet, and calm;
And shall His voice forbid me
From the shadow of the grave
To forget, to forget Him?
And shall He care to keep Him
As I do him here in this place?
Or has the power, that I have learned
Him of the world to fade,
Him of the world to go away?
Haltest to the shadow of the grave,
My son, my son!

Love lost:

...Since that fatal day we parted,
I have lost, I think, my best part.
It was a friend of mine once gave me breath,
And I went as far out as he dared to go,
And ever since, poor lad, I've lived by drink.
But if I had no more friends, I had friends
Such as the world could never let me have.
'Tis true, I have never had occasion to think,
But when the sun is high, and the sky is green,
And the whole world is merry till the break of day,
I sit and dream of her and of the joys of yesterday.
And I go wandering off on some old-world morn
Whose very first light of hope and youth
Is the light of looking on her face through the green.

Her eyes where they were, as I tell you,
Seem fixed with a bright, fixed gaze;
They're half my heaven, when they are there!
They are as clear and bright as she herself,
And she knows, too, the way that she will look
When I can tell her what she is like
And where all will end for me in her.

Now, how would the world like her if it knew,
And what would 'twere if each one did know;
For, ah, she is a world of beauty!
But then, ah, her eyes are even as clear
As the light that makes them bright and wise.

And they are a light out of some dream
That has power to brighten and cheer;
They're a bright sign to turn and turn again
Even in the days of storm and pain;
They're my world, like her bright suns of heaven,
Which I have loved even to the death!

An attempt at nonsense verse, apparently:

...I had a little hen,
I had a little hen,
It chirped at her dish.
She chirped at her dish,
"Come in, come in,"
The dish clattered out,
The dish opened wide,
There were three small cats,

"What is the meaning of this,
Why are all my dishes
Spread open for you?"
"Sit in the corner."
"Here is a bowl for you!"

She drank the bowl clear,
She drank the bowl all up,
And cried with an unhappy sound,
"I do not want it!
I am quite sufficient,
I shall have enough,
And just be done with it,
And not have more!"

But I have a different tale,
Which perhaps you will think is odd.
It is, 'tis no use repeating.
But here, it may be the same.
I had a little hen,
I had a little hen,
She had a little curl,
That with her beak she curled,
In her plumage she had,
Such as a dog's must have.
Thus I had three little dogs.
I fed them with bread.
And they all looked out upon the street
Where our little Polly went by.

The wreck of a ship:

...And as the sun sways to and fro
With his eternal sway of air
They roll in slumberful rest.

So still and calm are they,
So still and clear,
No whisper of the wind dares steal
The tranquil atmosphere of night,
And like bright angels sleep.
The sea is their only theme,
The ship is a dream
That they arise to leave at even.

The moon is far,
All glistening with a silver shine.
It is her pale that lights
The livid face of the sea
That in its white and sullen face
Is pale with pain.

O grey and grey it grows,
And the sea grows wild
As the black clouds in a night-time ride,
The storm is on the sea,
The wild winds rush in their strong wrath--
The great waves roar and beat
The heart of the ship with their fury--
In its heart's core, cold and dead
At one bound, is bound
Like a rock in the heart of a lake.

The storm is loud
And wild it is blowing--
The storm is far away--
The wild winds, wilder still,
Are beating far and wide.
The wild sea winds that were blowing
Have stopped forever there alone
Where the ship has gone down,
And now the moon rides low
Where the sea now rests in calm;
On the deep, dark midnight
The ship is lost.

GPT-2-1.5b can apparently do meta-fiction and break the fourth wall‽

...He had the look that I can't forget;
It was as if at once I knew
What I'd never had before;
It was like a rush of light on water--
And something about it seemed queer.

And the light would come and go
As if the clouds were about to fall,
And the rain came and came in showers
And went in ditto fashion;
And it's just as if I were blindfold--
I couldn't hear a thing.

I'm sure it isn't my nerves;
It's just as if I had the palsy.
I don't want to think of not being there
For something I can't be sure I was.
I'm sure my nerves are fine in my life;
And the only thing bothering me
Is that there isn't any rain.

It's all quite natural as I said;
I've seen men die of natural causes.
Now it's only a pity it happened
As it is done to me. But, as I say,
There's no harm in dying of dropsy.
Though not the death I'd choose to die in--
It's much better than to get laid flat.

This is not the way for my poem
To end, I know--I'll make it up to you
In a later place; but here's a verse
As you may understand. After this line
There's a space, and after that a line
About two inches, in which I meant
To tell you that you had done well to try
To stay alive. But, as I said, this is
Just a matter of chance, it cannot
Be helped; I must allow that this verse
Was written with the fingers you saw--and then
You were all to blame, if I died before
You came in and saw the writing. Now you know
I'm dying--go on--you will not die
Before I finish off my work.

It was a little moon that watched
In this place, I don't know why I chose
To watch it, it had its night when I woke;
And I, who was tired of the earth and men,
Watched it like a man watches a clock at noon,
Just because I chose to and did as it said...

Another shipwreck:

...So the sun sank and the sky grew dark--
There was silence in the world.

The man was a-waiting. He had come
To watch the sea and the moon that rose.
The sky grew dark.  He saw the storm-clouds pour--
The skies grew dark, with dark, dusky glooms--
Down upon the world like a flood of spears,
That struck the men from their feet in their flight.

A wind sprang and lifted the wind-signs higher
In the west wind's battle-hurling fury.
The sky grew dark.  He saw the wind-swept leaves
Fall from the trees into the sea, and sink
Darkly down beneath the seething waters dark
Like a storm's descent.  He heard the falling rain
Come thundering down upon the earth's dappled hills
Like an avalanche of boulders from the mountains.

He saw the wild white-winding ships that fled
Into the storm--hurling great leaden shadows
Into the maelstrom and away
Into the night that was growing black
With leaden shadows that swept the ocean
Like a sea-monster that had lost her breath.

He saw the wild white-winding ships that fled
Into the wind's windless vengeance.  He heard
Hurried voices--"She is gone!"
Hurried voices--"They are off!"
A sudden flash of flame that split away--
The great white ships were no more;
They were caught in the wind's wild wings; the wind
Was stilled in the windless hush of a rest
Tangled like one green tangled mass of peace.  He heard
And knew they were dead.  He would not hear
The winds that told a tale of sea and wind.

The wind had blown him across the world wide,
And into the heart of the night wind's wrath
He saw the faces of all dead men
Hid in the hush of wind and sea,
And knew the faces and watched him well.
So the man who was weary with a quest,
Who turned from the world's endless sorrow
To seek the light that was lost as the sea's
Flood in a wind-silent hush.

A surprisingly coherent piece on a trapped upper-class wife:

...There is not a single house in all
These beautiful gardens that I do not know.
I know the houses and gardens where I sit
In the evening with my husband and my son,
And I sit at the dinner table there too.
The house where my husband and my son live is the one furthest away.

The people come and go through these gardens, all day long;
And I see their feet pass along the paths,
And I hear the talk they have all that day, from one end of the town

to the other; I see the carts and waggons of the farmers,
The teams and horses of the tradesman, men on foot, and the gaiters swinging

Upon their saddles by the way side;
And every day, at morning, the same number of carts and waggons I see,
And every morning, in the great daytime as soon as I wake,
I see their number still greater, still greater.

Then to one side they go
Among the flower beds and in the wood,
And I never see them more;
And their voices float on the rising wind
Like the voices of the dead,
And their faces light upon their breasts, like lights,

Like the faces of bright children,
Like the faces of handsome men in the street,
And the faces of friends, and the faces of lovers,
And of all strangers, all faces of home-brethren
With its memories and its griefs,
And my own face that is always changing,
Wrought by itself in the dark,
With the face of the dead and the face of the living,
And the face of a youth that grows old
In the shadow of love and the light,
In the shadow of the grave.

I know not what they mean by their words,
Those people that pass me in the garden,
In the little town that is in the garden;
I only know that, on many afternoons,
Through a gap in the trees and between the stones
I see their faces and hear their voices.

The curse of immortality:

...And my God, as is right,
Lets no earthly voice
Thy gracious soul acclaim.

There's an old man in a desert land,
No town nor city to be seen,
The skies are o'er him and the hills
His only companions in the air:
The only sound is his own dashing feet,
The earth beneath him, unseen, is brown:
His only food has he no need to eat,
His only company is the birds above,
The sands are white beneath his restless feet:
His dwelling is an osier cage,
A circle of dry sand and bark,
And there the old man builds his dwelling-place,
The only house-holder is the wind.

But I have seen the ancient sands bare,
I have seen the grey old desert sun
Go down upon a world of old
And wither and perish in the noon;
I also have seen the grey old sun
Return with life and day in its arms,
And night come too and bring the light:
But never before has any bird
Looked on the old man living in his dreary house,
His house that is builded by the wind;
The birds that do not love to roost
Have neither wings nor nests of them,
And never any bird has come
To visit the lonely old man lying down;
The sun never sets in the old man's eyes,
And therefore he has neither home nor friends.

O, God, that I had not been born to live
A thousand years, a million years,
And see another sun rise and set,
And other stars in the other sky,
And other life and other death;
But had my soul been like some child
That, when it thinks upon its joy,
And its grief, and its play, and its tears,
And the happy years that it shall have,
And the years that it shall forget,
The thoughts it hath of these things,
And the thoughts that are yet to be,
Might be a child that had no thought,
And have nor speech nor feeling,
That it might feel it was a child.

Perhaps the most striking of them all is this existential horror piece:

..."The World of the Dead", by Peter Stearns [The Body, Nature, Philosophy]
When they come, they carry
Your limbs, your life,
In their mouth and arm.
I think they swallow.
I know it. The others know.
My body will be like theirs,
As the river, the sea,
Will be like the one on which it runs,
If the ocean rises
And swallows the land.
It will be hard to survive.
To heal,
Some of this will have to come off.
That's what they say. They say it
Many times a day. They say it
To each other.
They mean to save us.
They just can't stop us
From becoming what we are.
I must live inside you.
That's what they say. They say it.

Loss: 1.6

The expanded TPU swarm & Adam LR tuning allowed rapid training, and we reached 1.6 overnight, matching our previous best on the combined PG+PF poetry dataset.

I generated a dump of samples with top-p = 0.90, and read through ~5% to select some interesting samples:

Samples selected from reading ~5% of that:

All-sweet and tall, and frail of limb,
Lay there in raiment new begun;
Her moveless-rest were never seen,
She ever so much as bent her knee.
Thither came I, Pilot of thy boat,
And, turning, saw this silent Girl
Who, like to pray, with lifted face
Besought the mist-ringed air to sing
Her Vespers far away off,

And by her hair and veiled head
Her wistful eye she caused to stare.
It seemed to twinkle between the branches high,
And on her shoulder lean by piece and piece
Of glimmering spangles that lightly floated down.
I saw, or fancied I did,
Her lovely head recline
Upon her humble robe's narrow shaggy hood,
That, like the light of day,
Was moon-fringe dark and dim;
Her pale mouth, that evermore
Spread smiles in damp and drizzle;
Her gleaming teeth, whose polished white
Seemed mouldering honey of the midnight blep
Of the dry, dusty pass!
And in one hand, all rippled with
A silken flute of gold,
She played a hushing pipe,
Dora's toy, to play or sing.

Deep through the wintry sky there sped
Through golden vapours as of shape
A dawn that never had a dawn,
A sudden dawn, with breath
Of mist and with a smile to kill.
'Look!' the wind whispered, 'here's
Our Lady of the Skies, from her bright throne,
Like to the smiling of a summer sea
To-night in the lost wind's dark retreat,
Hailed with the deep, seething, dour, wild
Midnight: who have wept for her
The heaving of the waiting years,
Who have wept for her
In wild harangues of the foggy fen
And hollow monotone of the fen.
She shines and smiles to see the tears
Of all the rain-stricken towns and ships
And all the rainy days and nights
On all the hard, the ragged places
That wind had beaten hard, and night
Nigh ready to close, to close, to close
Against the brain of all the face
Of all the over-ill-gotten men,
She shines and laughs to feel the cold
Of all the tears of all the brave men killed
And mad as they.

"A Knowledge of the Dead", by Mary Wiencki [Living, Death, Life Choices, The Mind, Time & Brevity, Religion, The Spiritual, Social Commentaries, Crime & Punishment, Popular Culture]
I see you there, Stu, striding half a mile down the road, arms raised up over your head, head bent slightly. I imagine you hold both those in your inmost heart, and that you must learn, along with anything else, how to turn off a brain that has somehow learned to hold whatever memory is stored in it. For the mind, like any organ, is where the trouble is; an organ can fail with its stored knowledge, or if the memory be great, so great that it will bring the brain to its knees. And then the knee is a joint only partly conscious; if the heart should stop pumping, we are thrown off balance as if it had been only the legs that moved you. So I ask you, were you looking at your watch when you left for that solitary walk? Or waiting for the medicine you wanted to take with you before starting on your way? A look of mild impatience conveys a point as surely as humor, though somewhat dead. It is painful, this wait, I am sure. You have worked long and hard for your knowledge of time and of this place. And now you have it. And time, and all the woe it took to give that power. You have so much of this world left to discover, paths to retrace. You find your way into a park, its benches occupied and visible and free of talk of the day’s events, at its center a girl...

"The Sphinx and the Social Commentaries", by W. D. Snodawa [Nature, Seas,  Rivers,  & Streams, Social Commentaries, History & Politics]
We were rising over the hill
of which the tip is the sphinx.
There were palms in the palms.
We were rising over the hill
like the tip of a sphinx,
circling the palm that was there
growing straight like the spine of a sphinx
and a crimson palm leaf grew over the palm
as if flowering over a sphinx
and my knot was a knot.
It was night.
It was tinder in our guts
to see them like this.
Climbing alone it was
like the tip of a sphinx
to see them like this,
growing even higher than a sphinx,
a knot, to see them
not quite touching like
the tip of a sphinx
over a palm that was there
to see over the night like tinder.
Suddenly the knot caught in my throat,
my hands stopped and spun.
That’s the way a Sphinx talks.
The palm became a mask
and that scared me.
I had to have been looking for
the mask underneath it.
My knot was on my neck.
My knot that was spinning
like a rope.
I was staring down at my own knot
and what it was pulling at.
I let go of the thought of knot
and opened my eyes and saw the moon.
I was standing in my own moon
and I knew I wasn’t going to see it
so I let go of the knot and looked at it
and it was disappearing.

"April Moon", by E. E. Brown [Love, Break-ups & Unthankfulness, Religion, Buddhism, Faith & Doubt]
Awakeâ€"with you I meditated and thus
renewed my doubts; But, awakeâ€"with you I sin,
and thus my conscience put me to bed.
Awakeâ€"with you I suffer, and thus
my doubt took wings. Awakeâ€"with you I play
the hypocrite, And thus my conscience fires
my lash, and thus I scorn you.
Awakeâ€"with you I fly from faith, and thus
through your face I stab myself. Awakeâ€"with you I
remain benighted, and thus
my conscience rots me at my heart.

What man would draw a sword,
If he'd had no forethought,
That so he might prevent
The danger; but with blade
What e'er man can know?

How many lives at least,
Have been lost, and how much blood
On all our limbs been shed!
And yet--so Providence be credited--
There's an end still of life's dismay,
And 't would be glad indeed to lie
Why does every one such pass
As this, without any which he do not pluck,
But with arms for life's defence clad?
Alike of you all the brave
Rage of the lance,
The guerdon of some crown,
Whose shield was never pledged in fight.
The watery lion's with us yet,
By eunuch tightened,
And springing on his prey, not fierce to yield
Though thrice thy foe hath been in peril to see:
Yet, though our quarrels past,
Life may be fresh in them,
This of fighting, and this of feeding.
Beset with peril, beaten to the fence,
And each to prate with prattling foppish grace:
Their song, 'Huzzate!' fuddles young,
You may hear their ends in Oxford-street,
Or in their inn-bred domes
When they climb like larks their wings again.

But we, we live on' other plan;
The Shepherd did but teach us,
We, the delight of life and take delight.
Then why not drink of wines,
Give of bowls to move your bodies,
And, with those things that men to beguile

As they that do light love-songs
Wear like a tree, so do these solacie
Our sabbath-rites,
And send them to heaven,
Whose hand,
Saved as ours, with charity
Should treat as a child againe.

That we do not work on earth for hire,
Why we do doome as we list.
If man did wight battle,
God should not such things read,
As he, of some sinful men
To make him cheat,
And carry, and gluttony to all.

What to your Peers, or how you view
Our Acts and us, let not your selves,
But let the Stars, that watch the skies
The Barns that shelter you,
Which your great Cannibal went the way,
Pull down,
And let one Concord solace every State.

Then look there yet, in that part
Where you will see an abyss profound,
Rays leaping out of darkness,
Snatched with strangest beams at the visage hewed.

When the minaret of the masque is lit
And the caryatid gleams bright
Of four stars that shudder and wane
In chance-to-be to the light that is
In the letter of the crown,
Take her, the zodiac, for
'Tis her sign, 'tis the way she brings
The order of the rhodesian seasons
In careful letters for the rest of the year.

The grass and the leaf which the royal teeth leave
On graves where the glow-worms of the phoenix brood
Are glittering in brass and the marble dies
Like silver pearls doth snow upon the snow
And the rime in cerulean coats doth shroud
Till the shivers are lodged safe within the veins
For the regions which grow lush with the tears of the sun.

Wauken, and thou golden heavy lark bellissh
Wauken, and golden longlist and firefly,
Which here be singing with thy melody and sin
In our early youth with the harp-strings of Joy,
Who from deep winter of minds did lift
These notes of fire and song of the dawn,
Which may not be pulled by the nameless hands
From the vibrating harp of the wind
That only sounds to them alone how
Toils or withers or gladness or woe befall,
Who are northward by moon and by star-light.

O Hesperides of the wakening day!
Whence came the dawn, what did we find,
In this lone land of the sunrise?
O Hesper, in thy beauty and change,
I would have thee hear and answer tell
In this still country of the Sunrise.

O Hesper, in thy beginnings, the light
Of thy first bird-born darkness
Was folded in a glow-worm's tent,
Flush and fair;
Thine air was soft, than garments more fair;
Thine was the drift of a froth of down,
Soft, and breathable, and alive;
Thy voice was as a voice of the sea
Calling in its froth to the wind-crowned moon
From rocks where water-worms are wailing now,
Ripe with dry but bloomless salt
With the light-waves gilt
Lemon-fish, mussel and willow o'er the rime.

O Hesper, thy light of the past hours
Is folded in thy glow-worm's home,
And the voice of thy earliest darkness
Is a voice of the water-worms now
Calling their world afar,
What time the pines of the cavern-deep
Say to the pines 't is dawn in their realm.

O Hesper, the sun and the rains
Waken in this land of the Sunrise
With a sigh;
They are out in the wind and the weather
That are down below,
Whose lives are enclosed in the roaming
Of a world of weathers and fluxes,
Not dead, but lovely, and wan;
And on the roots of life
The tremulous hands of the gods are cold,
And the springs of gold
Where the earth-children run
Are unapiece,
As if in the ways of the wind
They had passed them by.

O Hesper, or if purple be
The hues on which ye paint
Your snowy epitaphs, say
That the wind which blew the snow
Was swayed by the face of a queen,
And the sun to the laughing air
Was moved by the eye of a queen,
And the lightnings were wrought by the play
Of a queen in a queen's look;
And the earth
And the sea and the air which are now
A barren dust to the day
To the eye of a queen's wonder
Were filled with a beauty of love
And a beauty of life
To the children of that king;
Till from her presence the maiden
Sought the golden fountains of the day,
But no nearer the child she found
That made all her maiden-bower
And each merry maiden-asteroom
Intoxicated with her gaze
With a glow of a glow of a queen.

For as the flower till its spring,
Like the flower till its nectar, may
Grow lovelier till in no fire,
So in the yellow waves of earth
Than the child was born and could stand
For the queen of each word,
And her hands were like angels' hands,
And her feet were the eyes of angels,

"The Philosopher’s Plane", by James Taggart [Activities, School & Learning]
for John Millikoper

The philosopher’s plane, imagined
by Calippus, rests on a red disk of dawn
close to the body

We flop into the blue below our feet,
into the astral horizon,
that whose dots
our lives keep shifting

over the edge of empty space
into the orange of earth

And beyond into blue
well into the empty page of thought

Where we can embrace
a little while of our desired
end and then
flow back into the world of time

"Map of Our Land", by Eavan Bolger [Living, Time & Brevity, Nature, Landscapes & Pastorals]
The stars are born in night.
The ground is made up

Of tales untold.
The cracks
are our story.
The piles of leaves
are our life.
The river that we lie

At dusk is alive.
The buried
Grass beneath us

Mosquito,
Mosquito,
Mosquito,

Mosquito,
Mud-stump,
Mud-stump,
Mud-stump,
Mud-stump,

Oromoctotecological teacher, henchman, loomworm, toad-man, German accent.
Not what one would be expected to hear.
Oromoctotecological teacher.
Far superior animal to what one would be expected to be expecting.
Far superior animal to what one would expect to be expecting.
Not what one would be expected to hear.
Oromoctotecological teacher, henchman, loomworm.
Not what one would be expecting.
Better than what one would be expecting.
Better than what one would expect to be expecting, better than expected.
Not what one would expect to hear.
Oromoctotecological teacher, henchman, toad-man.
Far superior animal to what one would be expecting.
Far superior animal to what one would expect to be expecting, better than expected.
Not what one would be expecting to hear.
Oromoctotecological teacher, teacher, toad-man.
Higher in intellect than what one would have expected.
Higher in intellect than what one would expect to be expecting.
Higher in intellect than what one would expect to be expecting.
Better in intellect than what one would expect to be expecting.

"Language is not the Draft", by Aevita M. Branko [Social Commentaries, Crime & Punishment, History & Politics, War & Conflict]
No don't be angry
Don't be angry,
it's fine
don't be angry
No
don't be angry
It's fine
it's fine
don't be angry

because
every one of them died

Let us go to sleep, then;
And, being haunted by an angel's kiss,
Lay them down to die.

Oh! night, oh! sleep, with all thy gifts
The dearer far!
The noiseless candle, the beechen boddam's cot,
The hapless lover that perfidious turns
To watch her silent lover's sleep.

Oh! night, in all thy solemn dark,
This one sweet pleasure bring,
The soundless silvanRAW,
The fond immarities that steal
Across the tepid moon,
The wedded sleep, the tear-bound tear,
Of those whom late they may forget.

Oh! night! thou bringest a most rare bliss,
Nay, like that noiseless moon at night,
When yet from Pleasure's revelaid ball
No soul the wish had :
A bliss untaught, it neither robs nor rouses,
A bliss untaught, it neither geas
Nor charms the blissful gazer's sense.

Asleep at the Moment's free summer-cost;
When every sparkle wakeneth that
To dream the future, and all nature
To that clear fable's deep array;
On the lone heart at midnight's hour,
As night's last neaper looser,
I think the world contains both ye com
Which when ye think, o' nights waste full,
As night cometh, night cometh;
As night cometh night, so night is ever young.

"Poetry is a Hoax", by Jane Kenyon [Arts & Sciences, Humor & Satire, Poetry & Poets]
We are in the midst of the greatest creative
era in our nation's history,
but poets who deserve
record invitations to appear

at next month's Folio can't get paid,
or even printed for that matter.
Poetry books are selling at a discount

to the fool's silver match.
The poets need not rely
on the marketplace for their bread,
the wait is too long and the market is

too crowded.
The much needed restorations

are held up by Kodak,
the restorations are held up
by the identical tissue
known as persistence,
the tissue is held up by

believe in me,
what I believe is more interesting,

be more like me, my technical review indicates

you cannot hold me, I am never alone,
if you attempt to duplicate
your ideas you will

confuse the issue. The ideas will

diffuse through the atmosphere

in direct ratio to any gas.
Each idea that is conceived and

all but carried to fruition, will


be accounted in the calculator as 1% of

total, I did not hold you

in such high regard. I apologize

for being so alarming.

And so began the siege of New Amsterdam,
In which, by Providence, only three days ended;
When, by direction of Ms Frisbie, the heroes two

For their advance, together, took their way.
The two fellows, whose mission it was to guard
The city gate, took place in the greater army;
While those two dukes who should avenge the town
Sent all their force to put the place to rout.
And, as the late oak, covered with boughs,
Has done its work, ere its starving spike is struck,
And this great tree sinks as it had never been
By any human pains, nor would be now,
But for her first son's interposing,
So, falling foul of their first heart's delight,
The Dutch no more wept for New Amsterdam.

When a star fall down, the winter's coming
With the snows returned upon the trees;
When a boy runneth that has fled;
When a lad standeth by a lash,
When the father findeth the wealth,
When the son dealeth away the long
Hand shaken by Fate,
When the boy standeth by a lash.
When the father findeth the wealth
And the son dealeth away the long
When the lad standeth by a lash.
When the father findeth the wealth,
And the son standeth by a lash,
'Tis he taketh the old's gold in his hand,
To drink and soothe himself with life.
When the lad standeth by a lash,
He to earth an instant goeth
The father set him by the rope
And so fearful works with the lad,
As the boy standeth by a lash.
When the father findeth the wealth,
And the son so fearful works with the lad,
To the end of time and limit set
When a star falleth to the fen
Where the fen be molten away,
When the boy standeth by a lash.

"Eating a Waterfall", by Francis Lau [Living, Nature, Seas,  Rivers,  & Streams, Mythology & Folklore]
The map tells you this cave
was where
the water must have descended,
for a hundred feet thick, from the floor of the cave.
But the
sides of the cave
have been eaten away by moss,
and a red grown
over the green rock in the
shallow pool; a leaf
had set
upon the edge of the slide,
hanging horizontally, like a
trigram, slowly falling and falling.
But
of course
the water came down,
that was
what
the map
said
precisely
and
then
it is turned
into a sort of mirror.
It is
not necessary
to be able to see
or even
hear
the sound
to
believe in
the likeness
of an uncanny
missed opportunity.
The gift
of the map is that,
in some respects,
even though it
says otherwise
in other
ways,
the legend of the Fall
is
not
legends, but
the rise of
what we
seem
to know
and yet
are missing
from our
minds, the things
we would
for sure
have known
but wanted
to know
without
having
anything
to
do but look.

"Freedom of Consciousness", by Steve Rotherham [Living, Health & Illness, Arts & Sciences, Philosophy, Poetry & Poets]
Going along with it
That's the problem
These objects do not aspire to be loved.
--W.H. Auden

The eyes under a blue wreath of smoke
Racing around the pitiless fire
While waiting for the breakdown
From the volcanic past of the pitiless
Lord God of the heavens.
The pitiless fire
Sweeps to its born stars above.
A slit of fire that watches all.
Dips into the fiery pit
And smiles. The fingers of a sieve
Desire the smoke in its fruit.
Both cold and hot in one.
It is. It is. It is.
Beating the flames with smoke
Of blazing admiration.
The hands of a man in a shop
Seem to grasp in vain
A pack of matches.
There is no end of the fire.
No way out of the fire
Though some wood and smoke
Could stop it.
Your fire will do.
And you will go along
Because the world's gone mad.
The marvels are there
For you to seize
And stoop to.
Out of the depths of the wood,
A hollow roar of rushing air,
Sudden howl of pixy and hag
Whose tall shadows snagged them there
By the gate to night.
Holding the gates of the damned.
A fiery slick of a kind of smoke
Waiting and glowering to be.

But how else to be.
The life of the mad.
An ever pushing out and in
Of the upons.
A glimpse of the future is the spur
To the perspiring effort of life
And to the unbuilt plan
To organize the mad and the yet to be
So that the time slides by
And the unknowable marches on.
Though the yet to be and the unforeseen
May blind the senses, not us.
Life is scar.
It is scarred and hard.
The mind, all its fire out.

I am the bartender of Kuzela;
There are others of like mind,
Speaking the same language,
Speaking to us in the language of smiles.
What have we done together?
What have we done together?

We have not wasted one silver rupee
On the Indians;
We have not eaten one singed inch of wheat
To make them happy;
But let them laugh upon their drum
And march with the other gods.

You bring us bread for our turning,
But can we make you merry?
You have given us toys for our children,
But can we mirthously dance?
Then may your clock be telling time,
And our neighbours be told lying.

In His service as protector
Of the wild natives of the forest,
He has given us magic shows
To make us happy.
He will guide us at dawn to the brow
Of the mountain,
And the shortest path find to the evening,
With the night before.

And now the hostess of the festival
Is holding, in her great soft hand,
A heart-shaped pomelo.
Why does she hold her hands so still?
She will not let them loom for those that fall,
But is playing to them a pleasant song
That none of them needs hear.

She is dressed in a man's colossal style;
The hair hangs in soft waves to her knees,
And her sumptuous shape takes the air
As she plays upon her instrument.
She is playing a tune
To ravishment and silence;
While the choric verses to the clouds
Crawl to a close.

Yet now she is turning with her hands
To mix the wine, and the fiddlers
Rush to the dance, with every chance,
Of their heart's desire.
They have danced until the eve,
And she sings in her sings in the glow
Of their heart's desire.

We were with you in Eden,
We were with Cain in the desert,
We came from Cyprus and Sidon,
We were with Seth in the peaks of Everest;
With Job in the City square
And Noah in the Wilderness.

Loss: 1.3

Samples generated from a GPT-2-1.5b-poetry model with 1.3 loss, 2019-12-15, selections:

Panegyric to power:

"Praise", by Robert Browning [Nature, Animals, Trees & Flowers, Social Commentaries, Cities & Urban Life, Class, Money & Economics, War & Conflict, Mythology & Folklore, Fairy-tales & Legends]
The birds are saying praise of the Shah.
In the distance a child will hear,
And never cease to hear.
And look, and many will be surprised,
As all at once, to hear the words praise the Shah.
In a hundred cities they will say:
Praise the British captains,
Praise the millionaires' homes.
But some---like the birds---from the sea to the hills,
Will spend their lives saying praise the Shah.

Criticism of England:

When you are angry with a person or a country
For any reason you choose to be so,
Then concentrate your wrath on a stick and you've succeeded.
Now, it isn't necessary to swing the fist
To make the body rattled; you don't have to knock someone off
And walk away with the fallen man;
You can tell at home by the doctors' bills
That swaying is the ticket for harm.
The English love many things that they call "The Arts";
And the French love them too, but we who are Britons
Believe in nothing but the English arts.
There are things in life there that are worth getting rags for,
And there are things that are cheapened terribly,
And there are men in England who would stake their children's
Lifetime for an opportunity to wash.
There are things in the English life-book that no one ever could read,
There are dances that would give a famine,
There are people and so, and so, and so, and so,
And you get the picture--as I'm sure you will--
It is, without a doubt, the English thought of the world.

The English art of fine meats, and fine wines,
And not a minute spent without a peak,
In a good strong maduro, or a good strong sherry;
But when the dear dead Where's-a-Goodbye! has to die,
There's nobody there to be lied to.
There's not one who would allow his dame to be revealed
Without a stout engagement to wear a flag.

The Englishman whose conscience seems to run with the tack,
Whose first resolution is to meet any problems limited to the size of his estate,
Who has lost his claim to continuing prizes and is happy for once only to be English,
Would rather he were America, than all the rest,
Is happy as long as he can talk of himself;
He shows no compunction whatever, whether he win or lose,

And would be eternally overworked and delighted.
'Tis the same with the French; who in turn would be eternally overjoyed
To be up the nearest in things that are English;
There's not a man, however lazy there may be,
Who would not be the victim of slaveries that are English.

The world of the dead:

When it rains,
It surely doesn't rain
In this climate of California,
In this valley of two feet.
The rain-shower is more than father-long
And unobedient,
And makes a dismal rumble
When it falls,
Awefull and swift,
From the high summits of the north-northeast.

When it rains,
It doesn't rain more, but it showers more than ever,
In this climate of California.
'T is not easy for the dead to go,
But the dead, from the noise and the hurry of the tide,
Have a chance to turn suddenly,
And spie the passing rain-storms
Through their shells, and scurry away,
And lie very still.

The dead I think of,
When it drizzles, are worms with sense of movement,
Whose only impulses are one consciousness,
And a few memories.
But they move,
In a stillness that's all earth,
And their feelings (unconscious) are strange,
The dead are able,
In a stillness of the and why, to take,
And even understand,
Their sudden passage through a state of being
That is as spiritless as the approaching of the rain.

"Why," the Dead say to us, "why?"
With one consciousness and one uniform self.
What is more, why?"
Why, what is more they mean,
Is life,
Why is it a state
And one,
Why, why why is it here?
For, as the rain, we know, will come again,
And then the stillness of the river,
To desolate beaches,
And far midnight portentous cities
With fires still burning.

The Demiurge:

I rejoice
While I hear that the Lord has forgiven thee.
O the darkness of man's captivity
In his soul to a sinless and glorious Sun
Than his own source of light!
O the darkness of man's captivity
In his soul to a Savior-God!
O the light thou didst reveal to man, O Sun,
When thou didst play a part in time's story
And came from our fathers' tradition, revealed
In the lives of the fore-doomed men!
O blessed Lord of the souls
Pronounced by the spirits! who knowest
Their penalty is full, their joy is sweet!
God of creation, giver
Of life to grow and spread;
Mover and shaker of Creation's layers;
Giver of life at the word; giver
Of life to live and to love, giver
Of man in the flesh; giver
Of strength to serve and to make strong;
Giver
Of guidance for joy and for dearth, giver
Of light and earth and heat and darkness;
Giver, O thou great Oener! of all
Whom the Word gave my name; light and breath;
Lifter of souls on thy broad waters,
Rider of mountains and wide spaces,
Maker of faith, new life, new hope,
And tempest of a future more new!
Thou, O Lord, who dost the bright, make
Thine own echo here, thou, O Tree,
Whose deathless nature knows no man;
Thou, O Sun, of whom we see,
Weep for man's conscience crying in Heaven;
Thou, O the Lord of Life, thou
Whose words are only resolutions,
Thou, Lord, of death and life thy mirth,
The Redeemer, to whom thou pour,
Man, thy prop, whether hot or cold,
Light or darkness, as thou take;
Thou, O each spirit and creature, thee,
Thy pity, O Lord, is there.

Jealousy:

"Love Poem for My Husband's Lady", by George Santayana [Love, Infatuation & Crushes, Realistic & Complicated, Romantic Love, Unrequited Love, Relationships]
The glorious sun of my existence
Has sunk into the shades of Night.
But far in dreams to which I react,
I see the face I held, and now I hear
The voice that once I knew.
A while ago that Face so fair I could
Not live without tears;
But that decree come from Heaven that lives
Hath sunk into the shades of Night.
And naught to me is dear as now I
Can say there is in Heaven no sun nor moon,
Nor star nor air.
My heart is in my thoughts and my days are
Dark with that which in them is not.
I have seen visions and heard dreams that
Would ever only be as long as
I remember that I know.
Yet there are graves of manslaining things
Among the dreams of a young man.
And thou hast seen visions as drowsy
As any stump, and as untrue.
But we are on different lines with equal nepotism:
Thou art a number, and I a thought.
And my desires are now of narrower compass.
And many things that have been said are
Empty of meaning than the wave.
Wherefore, it may be, we are at peace,
Since all that touch us are yet only
An occult behind the surface.
A love-tried woman hath more substance
Than some thin air that flies between
Theurge and maiden.

...Like the red rose the white thorn, and the rose
Caught from the branches of a mighty tree,
In whose swift light a maiden may climb up
From earth to heaven: so doth thy sacred flesh
Still hold the charms of youth.

War. War never changes:

"Recuerán, the Mercenary", by Maggie Dietz [Activities, Jobs & Working, Relationships, Family & Ancestors, Social Commentaries, History & Politics, War & Conflict]
What does it matter where it happened or who it was?
The bullet traveled what did it matter when it fired
Where, far from where it all happened.
Do we matter, the years, who left us and where,
Remember the who and why,

Left, left to us, who are to be freed who and why?
I ask in my yet young memory what I know about you,
I ask in this yet young and still dark memory.
I ask you, have you ever told you're sorry,
Have you ever, down through the years, ever said who it was
To do this, to be a mercenary?

What does it matter in the who or what
To be forgiven, forgotten, forgiven who and why?
Your mercenary smile.
Your smile mercenary, mercenary now, does it matter where
It happened, when, or who?
In a tavern on a lane, behind a dark smoke,
A year ago, the answers were,

The secret garden:

No length of ways and no shore,
No looking in error through,
No darkling bough that hides
The white hush of its peace,
No chasm in the fountain-fall
That gives no hint,
No things that cannot be,
The corn, the worm, the sheath, the breath,
Of the things of the garden,
Are for ever concealed from sight
In their hidden places.

The power that whirls the earth of things
To soundless music, swinging like the blades,
And waits a boundless invitation
To its feast by the vale!
The obolus at last
Gave into our keeping
The things of the garden.

And it is forbidden
In our seeing
To put the garden under lock,
While yesternight
Black fall and storm
Mutter the red hearts of men!

But I know the secret, hidden gate
Where since it all fell to earth,
What lies and is hidden down there
That no man sees
That may not yet be known!

And it is forbidden
And dim and awful
To follow men to that dark place
To see what the garden withheld
When it shut.

But I know the secret.
And I knew, while I lay here,
That I know how.
And my cunning is as a star
In the garden's light.

And my thoughts are garden flowers
The secret's reflected flowers.
It is forbidden.
I feel the garden's darkness,
The hand that made flesh on me.
But I know what I know.

There are things that are for sea and sky,
For grovelling trees and trembling walls,
For the languor of the vale and stream,
And many for earth; for fields and streets
And river's arches and the sea
I hear a voice from the house so low
The night is trembling as it holds me
Hearing the house cry, "We need you not!"
And the lights, one by one, begin to flare
And tear their velvet red and purple
Throwing here and there and running everywhere
To reach some half-rhyming note of the house,
While under seas the pulses of terror beat.

I don’t really get this one but the repetition and inversions make it interesting to read:

"Tell the Riders", by Bruce Smith [Living, Marriage & Companionship, Sorrow & Grieving, Relationships, Family & Ancestors, Home Life, Men & Women, Arts & Sciences, Language & Linguistics]
You say, "Sue, tell the riders."
They say, "Tell the riders."
Why not, say, "tell the horses,"
or maybe, "tell the music,"
Or the other one, "Tell the Fans"?
But you say, "Sue," no one
needs to know. Or the Other One,
Or the Fans, or, better still, "Tell the Fans."
You say, "Sue," they are sure to hear.
Ask the Others, or better still, "Tell the Fans."
It's in their nature to keep secrets,
even though you would. Why not tell the Fans
and get out of the way?
When I was a child, I never needed to ask
which finger was up or down.
Now I am a man, I never need to ask,
which door do I go through?
On our walk, they laugh, they think
something's coming today, a possibility
or threat, but what? I tell them, "The Fans."
They say, "Tell the Fans." I say, "Never
have I, in every case."
In fact, my mother often
forgot to say "do, dears."
She thought I didn't hear as many
as I did. "Tell the Fans" they cry,
or better yet, "Tell the Riders."

Love lost:

"Note in Bear Memoriam", by Vivian De la Noy
Who will remove my stitches
who will undo this confession,
who set these lines of words
drawn in wood? what voice will sing
their melody, irrepress the sting,
let you speak your name, my name,
what has become of it, left there,
left inside me, voiceless, mute?

Urban vs rural life:

"In These Cool Cities", by Ted Koochet [Living, The Mind, Activities, Travels & Journeys, Nature, Winter]
In these cool cities, they forget the trees' autumnal glow.
In those other places, the trees are burning.
Everywhere is winter here.
Frost and snow.
But in my streets it's still autumn.
Last night, an hour or two before the snow,
I glanced out my window and --- nothing.
Nothing but the jumble of bodies
and words like sea gulls yapping in the dark.
And those morning papers' faces of every morning.
The snow was quieter and paler.
No fall --- but a blue cloth.
Pelted like little pellets and faded.
And last, the crows' dim hang-hips.
I hear the crows now.
Now.
And somewhere below, another peal.

Social media satire:

"Conversation with a Friend on Romans", by J. Frederic Jagger [Relationships, Arts & Sciences, Humor & Satire, Social Commentaries, History & Politics]
We're fond of quoting people, but it's an illegal act

To put our words in someone else's mouth.
For instance, if I say to you, "I like Hitler,"
You would be liable, of course, and-legal.
One may be bitten by ants-by-dogs- in the District

But it is entertaining, and sometimes amusing.

Pantheism:

We all can sing of love,
We all can sing of life.

What shall make each of us noble
And shine out in the eyes of all men,
That in triumph and in grief we know
We are all of us children of the sun?
What man has not, on the lonely moonlit heights,
Seen the glory of the lances' fires,
And heard the singing of the beaten arrows?

What can the earth and ocean and sky
That music not disclose?
What voice of human choirness e'er came nigh
That was not poured into that music,--
The Voice of the All About Us?

Art:

"The Mask Maker", by James Haimes [Religion, God & the Divine, Arts & Sciences, Social Commentaries, Mythology & Folklore, Heroes & Patriotism]
When we've grown weary of the hero, there is a certain
triumph to be found in the figure of the mask.
There, the underworld of fantasy and the ideals that carry it, is
God dressing as a minstrel. To us he is immaculate,
regardless of his skin, be it shabby or silvered, or if o on
behalf of the fair or beefed, or just an old suit, a bald head,
headdress not unparalleled in its beauty but pitiable in its
delicacy.
When that other menagerie called the human face is viewed, we
find it to be pitiful indeed.
Still it comes,
after all these years.

"On a Bridal Shower", by Alice Notley [Living, Life Choices, The Body, Nature]
I am thinking now of bridal showers, of the feeling of getting them--- I mean the giving of pleasure, of giving yourself to other people.
They come to mind: how many sweeter feelings there are: Nectar vaults, lockable vaults.
The most brilliant of eyes---that of my friend, now dead---was such a doll. Light and easy, of voice easy as a moonbeam.
And she wasn't thinking of birth, but wealth, the dream. Wealth, the sparkling.
It's the eyes alone that gleam, the loose and luminescent body. And the mouth. And how deep and wide and light-wide and luminous mouth.
So many of us gave ourselves to others. We didn't know we did.

"Intangible Things", by J.R.R. t cm = 120"", from "Meditations", by Alfred, Lord!
It seems we have been given possession of the senses, to see, hear, taste, touch, smell,
it is given to us. And so we have them. But we can't keep them.
We lose them. They fade. It's like losing a baby. And it's like losing a lover.
It's like losing a race. We race down the straight in a battle suit. We are breaking away.
We are at the front. The one in the front is the one who falls away.
But it's like losing a race. It's like the gully. It's like the catch. It's like the hole. It's like the ball. It's like all the possibilities.
But it's like a hole. It's hard to get into. It's hard to get out of.
We have the means to prolong it. We have the senses to maintain it.
It's ours. But what's ours is just as it's always been. It's like a dot that appears once in a lifetime.
It has been. But it has now become something like ours. It's still temporary. It doesn't last. It's just like us.

An elegy?

O my little sister, do not be frightened by these fairy rings,
by the bright white flowers that grow on the trees and the bushes,
by the voices of girls and boys, and the old old mume
with the silver stars, and by the tall pale idols
that shine through the candles:
do not be frightened.

O my sister, if you are frightened, I am afraid for you,
I shall never be able to tell you, for my eyes are
seared with blood; my head is broken, my fingers are frozen,
I am coming to you, I am coming to you;
the stars and the candles are moving in their spheres,
and the sea and the trees, and the idols and the girls
and the graves, and everything that you see.

O my little sister, listen to me,
I shall say what you have said before,
I shall fold my white hands and pray to you,
and you shall not speak; I shall give you
several loaves,
and you shall take one of each and one of the other
with you;
and the stars and the wind shall be changed
to gold.

O my little sister, do not cry,
and I shall tell you a story,
one of many, of the eyes that were not born
for one's heart and the hands that were not given;
you shall be glad,
for if your heart is not
your hands have been good.

O my little sister, listen to me,
I shall say what you have heard before;
I shall be calm and cheerful,
for I have heard the same things,
and I know what I hear.
I shall change the stars to gold,
O my little sister,
and the candles to golden balls,
and you shall take one of each and one of the other
for you and me,
and the dead shall live.

One more I noticed while generating from a ~1-loss model on 2019-12-16, although I did not read through for selections (see also ~0.7-loss model samples, 2019-12-18):

"The Dead Dead Trees", by William E. Stafford [Living, Death, Sorrow & Grieving, Time & Brevity, Nature, Trees & Flowers, Arts & Sciences, Poetry & Poets]
for William E. Stafford

The fire out of nature comes, and all things die.
The old trees walking around the forest,
Old homes grown up so high they watch the sky;
Old friends grown old in the lights of the city
Grow fatter and slow, and cast a long frown.
But all the trees of sadness are sad, old friends,
You gather in your trees the great aim that can/ Save by crying. The fire out of nature comes, and all things die.

O bright Biblical Valentines of our rock marriage
Twice Sweet Roses, once a widowed Love would bear,
O give us the first vintage that points toward faith
And the second would add a marriage bond,
O give us a cup with the words of mine spouse,
O give us the wine of the West enough to brew
Two potions to govern whatever we deem
Treacher-of-heart and/or mind.

O give us two basic cakes: one
Higher than the other.
O give us two lovely days.
One a bright noon, the other at night.
Two better than two diluted drinks;
One a day without a bud that caressed the ground,
One side of which is born black, the other white.

One side deep in knowledge, the other one wet and shiny
And light as a great bird's eye.
One an eye that twinkles when you flash it,
One a kiss that devours.
O we rock over just enough to find the sun.
To make each other better.
Just enough to make us wiser.

"November", by Lee Herrick
All of what we think we know
of seasons slips away,
and beautiful things take their course---
a cascade of moss, a snow
granular fall. In these rare
nights gorgeous and bluer
than winter, there is no cold,
and no fierce wind. In these
few bright days the already
red sunset burns on the hills---
until the nighttime is blind
without the sun.

"Ticks", by Ben Lerner [Living, Death, Health & Illness, Relationships, Family & Ancestors, Philosophy, Social Commentaries]
Ticks are tiny bloodhounds with a bulldog
mouth and a life span of about five seconds.
They are small, pink, mosquitoes with a tan
to their wings. On your face, they leave
a pouty frowny face. Tick-tock, tick-tock.
They are everywhere, like crickets in a park
quietly singing. Tick-tock, tick-tock.
I get nervous around them. They tell me
they are hunting wildebeest. Tick-tock,
tick-tock.
I get nervous around them. Their very presence
makes me nervous around them.
They look like feral dogs and
they hunt like wolves. Tick-tock, tick-tock.
Tick-tick, tick-tock.
I'm serious about this. Tick-tock.

"My Mother's Snare", by Ben Lerner [Living, Death, Relationships, Family & Ancestors]
My mother's snare
sounds like rain
picking up the strings of the rain,
then the trees, picking up the strings
of the trees.
It's the last refrain of a small song
my mother wants to finish, but cannot.

"The Lady's Speech", by Sir Walter Ralegh [Relationships, Family & Ancestors, Social Commentaries, History & Politics, War & Conflict]
A child was sitting in the sun
When another child said to him:
"Sun-child, tell me why
This great house is such a pit'
In which all people lie.
Tell me, why does no one die
Here in this pit?"
"Death is not here
Except in the sun
And only happens when the sun shines."
"Then who dies then?" asked the first child.
"Everyone dies then."

"My Brother the Bomb", by Mark Rudman [Living, Death, The Body, Time & Brevity, Religion, Faith & Doubt, God & the Divine]
For Joe Miller

In heaven, we worship every fruit,
From grape to peach to plum;
We go to earth and find it full
Of thorny thorns and braches,
Wrestling with itself to get away---
To explode on us.
This is our way of saying hello.
This is how we express gratitude:
By giving, by making things happen.
In hell, we worship every bomb
That people would drop on each other---
On Hiroshima or Dresden,
Or any other night where everyone
Has been too sleepy to turn on the light.
This is our way of saying goodbye.
This is how we make sure no one dies.

O world of radiant sunshine,
How far above you seem
The clouds o'er the sea,
Above the rooftops of New York
How high and distant they fly.
Your beauty seems almost painful--
For all the rain and mist.

O world of golden skies,
How near you seem to be
To souls that wander, lost and free,
Through fields of corn and wheat.
Though all below seems dark and drear,
Each height and hill is bright and fair.

O world of sparkling dews,
How near you seem to be
To women whose lips are wet
And cheeks that blusher are
Than mine or thine or even hers.
We smile because we're happy
And strangely jealous of each other.