Betancourt: higher-order autodiff

Just stumbled upon an open tab with Betancourt's "Geometric theory of higher order automatic differentiation" which I started reading the new year night but quickly got distracted from. I remember my first feeling was that it's slightly more verbose than actually needed and that I'd have used some different wordings. While this might be true, I'm finding the survey in its intro very clear and explaining. I must remind my imaginary interlocutor that I have not as of yet went through any course of differential geometry, only skimmed some textbooks and wikis. I've been really struggling. Mainly because of terrible notation and language established by physicists, I believe, and employed in most classic texts. There are exceptions of course. Some of seemingly good texts include e.g. works of Lee. I think I would've solved my problems if I read carefully Lee's monographs and walked through exercises therein. I'm not yet ready to make this effort (not ready to do anything at all).

As for higher-order differential geometric structures, I have only encountered jets when reading Arnold's "lectures on PDEs", where they were in fact treated (if my memory doesn't deceive me) as "things" that appear in Taylor expansions, without actually specifying their "datatypes".

Now, here's what I actually wanted to remember when I started typing this note: a good survey rapidly introducing principal concepts before verbose and detailed body of a text makes a lot of difference. Lee, say, pours on you quite an amount of information that makes use of terms that haven't been yet made concrete. And that information might give you a good intuition if you already got some very basic framework of concepts and notations to add new nodes and connections to. Betancourt on the other hand throws "pullbacks" and "pushforwards" at you in the very beginning. He throws them pretty concretized, almost tangible, in the sense that he defines domains and ranges of the mappings, and essential properties of their actions. He doesn't spend too much time on it, doesn't overload the reader with questions like existence. These questions are important later for rigorous analysis, but not for sketching the initial map of the field, not for initial understanding of connections to other concepts.

Brain is a terribly lazy thing. At least mine is. The art of writing (when the purpose of writing is to explain a subject and convey a message) is in hacking reader's brain so as to leave it no chance to object comprehending the message. That's an obvious thing that I always knew too. Yet I tend to forget this when actually writing.

Prerequisites for getting asleep

So, it's 6am again. Yet another night without sleep. I sort of figured out already the reason I don't go to bed, but finding precise statement is always a separate and more difficult problem. So it just occured to me, that basically I prohibit sleeping without having decided what I'm going to do next day. I need to have at least some time of the next day to be 100% occupied. Some deadline coming, some meeting scheduled, anything. Something that requires my action and something that affects other people. Because when there isn't anything involving the outer world I always wake with that single thought in my head: "no reason to be awake".

"No reason to be awake". This thought is really uncomfortable. More like frightening. It is also consequenced either by my refusal to wake up or by me waking and doing something stupid. In the first case I would sleep some longer and then still get up, because too long a sleep makes my body ache. In the second... I usually lose something as well. So this thought is to be avoided.

And it's not that I've got nothing to do, though it often lookts that way. I got lots of things to do. Got lots of ideas to try out, got some projects, got some responsibilities. And yet, right now I'm not working on any of these. Haven't been for quite some time. I'm walking on the streets. I'm drinking. Talking to strangers.

So, right now, as I'm writing this, I ask myself: "Thibaut gave me some quite specific problems; there are quite deterministic steps to be taken to begin the work; why are you not doing it?"

I wouldn't know the reason, but at this moment I'm thinking it is because I don't see what possible impact it could make. So, when somebody in the Skoltech chat asks a stupid question about a lecture, I immediately switch my all attention to it, because I know what impact helping another student would make: it might help another person get the concept and switch to next problem, saving time and effort. It's not much, but it affects something outside. It's not limited just to myself. The effect's almost immediate too, but I don't know if that matters.

I'm not saying I care about other people or anything. I'm just observing facts: naive questions drag my attention just like that, and I'm really failing to begin implementing any of enterprises I'm dreaming of. The only difference between my own ideas and other people's little problems (the only that comes to mind) is that my ideas are limited to my mind and other people's problems exist outside. So it must be that I care about impact and other people after all. Unconsciously, against my will. But I apparently do.

A truth

Back to watching TVs in the background throwing my time away. Yea, it is reprehensible and all that, but... just take a look at what cool things you sometime may find:

House MD, S05E03: Stop saying "a truth". There's only one truth

Linguistic beauty of English theatre

One of my best discoveries of past two years are NTLive recordings of theatre plays. I've been feeling lately as if they were that only thing keeping me going.

Julius Caesar staged in London Bridge, Spring 2018

Bridge's "Julius Caesar" is my favorite of all performances. When you enter the show you get confused for first couple minutes: is it a Shakespearian play or some sort of rock fest? Now Julius Caesar is in my opinion a story about a crowd and madness before anything else. And Bridge's immersive setting plays this point out nicely: the play literally happens right in the midst of spectators, some just watching, some chilling with beer. Wish so much I were there!

Cassius and Brutus:


Brutus, I do observe you now of late: I have not from your eyes that gentleness and show of love as I was always wont to have


Tell me good Brutus, can you see your face?


No, Cassius, for the eye sees not itself but by reflection, by some other thing.


't is just... and very much lamented, Brutus, that you have no such mirrors as will turn your hidden worthiness into your eye.


Into what dangers would you lead me, Cassius, that you would have me seek into myself for that which is not in me?


Therefore, good Brutus, be prepared to hear: and since you know you cannot see yourself but by reflection, I, your glass, will modestly discover to yourself that of yourself which you yet know not of.


... What is it that you would impart toward me? If it be aught toward the general good, set Honour in one eye an' death i' the other an' I will look in both indifferently! For let the gods so speed me as I love the name of Honour more than I fear death.


...I know not what you an' other men think of this life, but as for my single self:

I had as lief not be as live to be in awe of such a thing as I myself!

I was born free -- as Caesar. So were you. We both have fed as well, and we both can endure winter's cold as well as he. For once upon a raw and gusty day, troubled Tiber chafting with her shores, Caesar said to me:

"Darest thou, Cassius, now, leap into this angry flood an' swim to yonder point?"

Upon the word, accoutred as I was, I plunged in and bade him follow! Which indeed he did. The torrent roared! An' we did buffet it with lusty snews throwing it aside. Stemming it with hearts of controversy. But ere we could arrive the point proposed, Caesar cried: "help me, Cassius, or I sink!"

An' I as Aeneas -- our great ancestor -- did from the flames of Troy upon his shoulder the old Anchises bear, so from the waves of Tiber did I that tired Caesar!

And this man is now become a God! An' Cassius is a wretched creature who must bend his body if Caesar carelessly but nod on him!



The fault dear Brutus is not in our stars but in ourselves that we are underlings



Now, in names of all the gods at once, upon what meat doth this our Caesar feed, that he is grown1 so great? Age, thou art shamed!

Note how is grown works apparently as a filtration, removing information about preceding process or, more precisely, emphasising the lack of such information. An example of what Michael Polyakov has described in his blog.

Rome, thou hast lost the breed of noble bloods! When went there by an age, since the great flood, but it was famed with more than with one man? When could they say till now, that talked of Rome, that her wide walls encompassed but one man?

Now is it Rome indeed and room enough, when there is in it but one only man?




Till then, my noble friend, chew upon this:

Brutus had rather be a villager, than to repute himself a son of Rome under these hard conditions as this time is like to lay upon us.

Caesar to Antony, about Cassius:

Let me have men about me, that are fat! Your Cassius has lean an' angry look, she reads too much, such ones are dangerous! ... Would she were fatter!

Caesar, Calpurnia, and Decius Brutus


Say he is sick


Shall Caesar tell a lie? Have I in conquest stretch'd mine arm so far, to be afraid to tell graybeards truth? Decius go tell them Caesar will not come.


Most mighty Caesar, let me know at least some cause, Lest2345 I be laught at when I tell them so.


The cause is in my will: I will not come! That is enough to satisfy the senate!

Antony mourning for Caesar:

Had I as many eyes as thou hast wounds, weeping as fast as they stream forth thy blood, it would become me better than to close in terms of friendship with thine enemies.

Oh, pardon me, thou bleeding piece of earth, that I am meek an' gentle with these butchers. ... Over thy wounds now do I prophesy, which like dumb mouths do ope' their ruby lips, to beg the voice and utterance of my tongue, a curse shall light upon the limbs of men; ... And Caesar's spirit, ranging for revenge, with Ate by his side come hot from hell, shall in these confines with a monarch's voice cry "Havoc!" and let slip the dogs of war!

Madness of King George with Mark Gatiss

Willis and George

Doctor Willis:

You must behave

King George:

Must? Whose must is this: your must our my must?

Also King George:

I'm the king, I am not told! I am the verb, sir, I'm not the object.

Hamlet with Cumberbatch

The first play I've watched. And if there was any that really gave me chills this is the one! Deep, consuming darkness and Cumberbatch's amazing accent.

Here we have an Evil King:

My offence is rank! It smells to heaven!


My words fly up, my thoughts remain below. Words without thoughts never to heaven go!

Do it, England! For like a heptic in my blood he rages!

A young man who wasn't quite ready for what should had happened:

Smiling, damned villain! My tables, "meet it is" -- I set it down -- "that one may smile, and smile, and be a villain!"

How weary, stale, flat an' unprofitable seem to me all the uses of this world...


Foul deeds will rise, though all the earth o'erwhelm them, to men's eyes!

And many other beautiful characters and lines, including popular "whether it is nobler in the mind" and "poor Yorick!"

But! We are, of course, primarily interested in what confuse commoner's grammatical intuition. I don't remember many of these but it must be because I've re-watched that production about ten times already...

  • Begin with something simple:

    Who's there?

    Nay, answer me: stand, an' unfold yourself!

    Not really confusing, but curious.

  • Almost immediately follows:

    't is struck twelve: get thee to a bed, Francisco.

    This form will later be repeated by Hamlet addressing Ophelia:

    Get thee to a nunnery -- go!

  • Gertrude:


    Thou knowest 't is common: all that lives must die, passing through nature -- to eternity.


    Aye, madam, it is common.


    If it be, why seems it so particular with thee?


    Seems, madam?! Nay -- it is! I know not "seems"!

    This is actualy a part of "oh that this too too sullied flesh would melt" and "how weary, stale, flat, and unprofitable" speech all of which -- including conversation with King Claudius -- is a must-read (must-hear)!


> To ears of flesh and blood. List, list, O, list!

(if not obvious: it's as in "listen" but used with "thou")


> The play's the thing wherein I'll catch the conscience of the King!
  • Someone (what? it may be a spoiler), after stabbing by mistake a wrong person:

    I took thee for thy better!

  • You can also watch how questions without an auxilliary verb leave much more of an impression:

    This was your husband. Look you now what follows: Here is your husband; like a mildewed ear


    Have. You. Eyes?!

    Or this by Rosencrantz, Hamlet's pal:

    Take you me for a sponge, my lord?

  • Just a play of words:

    Now, Hamlet, where's Polonius?

    At supper.

    At supper? Where?!

    Not where eats, but where he's eaten.

    And more:

    Where is Polonius?!

    In heaven! Send hither to see: if your messenger find him not there, seek him i' the other place yourself!

  • Some "many a":


    How does your honour for this many a day?

  • A little bit of had-conditionals:

    You that look pale and tremble at this chance, that are but mutes or audience to this act, had I but time -- as this fell sergeant, death, is strict in his arrest -- Oh, I could tell you.

  • Some rather rare curses:

    What is he, that builds stronger than either the mason, the shipwright, or the carpenter?


    Marry, now I can tell.

    To it!


    Mass, I cannot tell!

  • Some cute valediction:

    "...he, that thou knowest thine, HAMLET"

Contless they are: you better just watch it!

Macbeth -- the Scottish play

I would like Macbeth more than all the others had it been staged as a well-thought and consistent performance. So far all the productions I've seen are really bad at details -- NATO uniform, my Gosh -- and definitely don't cast nearly enough of that black despair unto you that they should cast. The sadest thing is how the witches are played -- shrieking with these stupid childish voices. Most like a joke.

Yet as compilations of separate scenes...

There was this production that was trying to mimic a horror movie (which made most of it look like a joke) -- I can't recall exactly whose. It had an outstanding "Knock-knock" scene: as Seyton performed Hell's gate keeper the dead were falling out of windows and stoves.

Be sure to watch "Come you, spirits! Un-sex me here, and make thick my blood" scene as well as "tomorrow, and to-morrow, and to-morrow". Some creeping rhythm there.

King Lear

Another play that I think must fill spectator with fear but does not as it's usually not played well enough.

I still like "blow winds, and crack your cheeks! rage! blow!" lines and the Fool's songs as performed in some of Globe's productions (the one where both Kent and Fool are gals).

Ian McKellen's performance is remarkably boring, but got some very cool (almost psychodelic) pictures (e.g. the ones with the King and blinded Gloucester with a ragged wall in background, if I should remember).

Much Ado About Nothing

Watched the one staged in Shakespeare's Globe. A very inspiring play about disdain turning into love full of merry songs and dances.

The Hollow Crown

A (probably low-cost) TV series that begins with Richard II and ends I think with Henry the V but may be even later.

I particularly like Richard the third with Cumberbatch playing the main villain.

Now is the winter of our discontent made glorious summer by this sun of York: and all the clouds that lour'd upon our house in deep bosom of the ocean buried.


Why, I, in this weak piping time of peace have no delight to pass away the time, unless to spy my shadow in the sun and descant on mine own deformity:

And therefore, since I cannot prove a lover, to entertain these fair well-spoken days, I am determined to prove a villain and hate the idle pleasures of these days.

Henry the fifth is also shot well though I still don't know the name of the actor.

Once more onto the breach, dear friends! Once more!

Romeo and Juliet

Watched that teenager version and it was quite cool.

What's in name? That, which we call a rose, is by any other name as fair.

  1. Michael Polyakov, "How tenses work" 





Eric Moulines' mini-course

I was a fool enough to miss most of Eric's course. Well, just one lecture out of two, technically. But still. Though it wasn't much of my fault either: that Saturday had been reserved long time ago.

So, I listened to his talk last Wednesday which wasn't exactly impressive, then missed the first lecture on MC, and today finally arrived for his last lecture which was really cool. He's a great reader!

Now, I'm to remind you that I don't know much about measures and probabilities, it's not my field. So most of the stuff that I found cool might in fact be quite trivial. But here are things that were new to me and seemed cool:

  • Hamiltonian paths are closed. For this reason when doing HMC we got to be taking random lengths lest our iterations end up just where they begin
  • Total Variation is actually a norm on measures which is dual to -norm on functions! Really lovely. And quite useful, because I couldn't persuade myself to try and really read the definition of TV for along time now. You know, analytic definitions always make you feel disrespected -- it is as if the author didn't care enough to pre-process his idea into a form convenient enough to be taken as an apparent geometrical truth.
  • There exists a coupling formulation for TV: For measures there exist -valued random variables (on some prob. space) such that
  • We've also covered Dobrushin coefficient.
  • We've even applied Banach's fixed-point theorem! Felt like I was home. Not all this stochastic-differential-rubbish that they don't care to define signatures of!

Getting started with GANs: Part 1, getting access to GPUs

Here's how the things are now: I finally talked to Thibaut and he agreed to supervise me; probably we'll be working on something related to barycentre or curvature in geodesic spaces, or whatever (we've discussed some possible problems but it's not that we've agreed on something very specific yet); that's all very fascinating mathematical stuff but sooner or later I got to catch up with the real world; so one of the things I hear most often of and related to optimal transport distances are WGANs and it seems reasonable to begin my acquiatance with what's outside with them.

The nuance about this my endeavour is that I've never in my life actually trained even a CNN. Always felt like it's something trivial and not worth knowing the details of...

As @ferres suggested, before getting all into WGANs it's worth preparing a baseline. So I'll first get through Goodfellows old GANs.

I'll be using the cheapest GPU-powered AWS instance out there:

It's nothing too fancy, just better than a laptop with integrated graphics:

I'm covering it with the studentpack promo worth $100.

...and it won't launch:

Spot instance wouldn't launch either -- "no capacity". So requesting a limit increase.

Notation for DEs in Hilbert spaces

Look guys, I appreciate and understand that Hilbert are self-dual, but if you use two different notations for differentiation in a single equation when you could've used just one -- you got to really mean it.

Look at this:

Wherefore do you write so many symbols if could've written less and thus avoided being a liar? Take for a Hilbert space in discussion, the left hand site is the derivative -- a row-vector , and the RHS is a column -- same derivative encoded as a vector.

\( \mu(\mathrm{d}x) \) and \( \mathrm{d}\mu(x) \)

't is known that if you begin from Lebesgue point of view you naturally end up with notation where can be thought of as an infitesimal set; Riemann on the other hand leads you to which stands for "change of measure". This is all of course a birdish language which doesn't define anything for real and does not help computation.

Now, I really hate all these limit-centric notations. In smooth and bounded non-random finite-dimensional Euclidean world (in other words in any naive calculus) we most elegantly get rid of it, saying that is a differential form, that is it maps each into a multilinear function -- a tangent, a multilinear approximation of the integral curve. Then even though the action of the linear operator may involve limits in its definition -- which is perfectly fine -- our is a pretty understandable object that has specific type. The integral curve is naturally characterized as the one with given initial value and tangents at each point. This in a rigorous way captures the original intuition of operating with "infinitesimal increments". The bird-language phrasing "for every small enough a " actually means that we define a linear map that turns every into approximation of .

But why -- it feels like I fail to construct even for myself an analogous interpretation for all these or .

The naive way to describe the latter is to say simply that it is a random linear operator -- the derivative of for any fixed outcome .

Yet with the set... isn't very linear a space and it isn't about linearity neither. Also this notation isn't going in line with the main idea -- that we're actually splitting the image space instead of the domain.

Porto and curvature

To-day (or more precisely the day before this night) was: a bottle of Chateaux (Merlot, Caubernet-Sauvignon), one of Don't-remember-what, two shots of Sandeman vintage, and a glass of Sandeman tawny.

I found all of them sweet to taste rather than dry which they in fact are. Portos were too sweet to me (even tawny one), most like Cohors. But when limited to a single glass methinks they make a very noble drink for our cold times.

I also learned about some really cool cake: P\`avlova (thanks to they-will-understand-I-am-reffering-to-them)

With three good Portos Villani's lectures start looking especially reasonable and charming -- feels like you understand the world! I'm now watching this one: It got quite some praise in some medium post that I found in some tweet. Now what I mean by "charming" is for example these stories about "Kantorovich's duality theorem being considered a very capitalistic theorem". TLDR: "capitalistic" is actually the strong duality which tells in the case considered that trying to minimize over constraint set by yourself gives the same result as letting a hired man maximize his profit. But him mentioning "(about Monge and Kantorovich) both were extremely precautious students, and both of them were proffesors at the age of 22" drives me back into depression. I'm two years older than Galois was when he got shot, and I've done thus far nothing.

Oh, I also finally had some good runs: night, forest, and speed-mad doge pulling forward. Long gone feeling. Running e-track at dorm can't even be compared.