Convolution is a linear operation; in the case of images, you can view it as a multiplication with a doubly block circulant matrix. I can't see any barriers to hybrid approaches here, though it seems difficult to avoid using backpropagation for credit assignment within the convolutional layers.

Re: significance, how about their application in regression: https://arxiv.org/abs/2006.05964 ? Or in contextual bandits: https://arxiv.org/abs/2002.11611 ?

Disclaimer: I am one of the authors (Joel).

Wow, that G-GLN paper is really new. Thanks for sharing it :)

My first impression is that this will be very challenging to use for many of the people currently using AI in practice, because of the requirement to have a convex and gaussian-distributed result.

For example, I currently work on optical flow and the loss functions are usually very jumpy and usually only convex within a few pixels around the correct result. I have seen plenty of modeling errors in optical flow SOTA papers, for example casting a boolean occlusion term so that it can be added to the loss (Won't work, no gradient). I have also seen how strongly authors struggle with the irregularities of their loss function and issues with convergence, for example by fixing all random initialization to a seed and then parameter-scanning on that value (Very expensive).

Notable examples of the difficulties faced would be https://arxiv.org/abs/1904.04998 which I have never seen converge from random initialization or https://arxiv.org/abs/1711.07837 which diverges without supervised pre-training.

Also, while many people use the gaussian-distributed euclidean norm of the difference between prediction and groundtruth as their main loss term, there has been a lot of discussions recently if that is a good idea, because it forces the neural network to represent uncertainty with blur. But optical flow tends to have sharp edges where objects end.

Combined, that means the problems that I work on usually do not have a gaussian-distributed loss and usually are irregular and never convex, so I'm not sure if I could use G-GLN.

But the application to contextual bandits looks VERY interesting to me :)

I see great potential in using conv layers as pre-compression and then applying decision trees on the resulting intermediate representation.

What I previously did for object segmentation was to sample a random but fixed set of feature pair differences and use the signs as bit flags. I then trained the decision trees on those bits to predict the boundary shape around that pixel. I got the general idea from this paper: https://arxiv.org/abs/1406.5549

But it sounds like moving from difference bits to halfspaces and from linear regression to GLN, your paper "Online Learning in Contextual Bandits using Gated Linear Networks" could greatly improve on that. I'd be curious to see how those bandits do on BSDS500.

BTW, are you aware of any discussion groups for AI image processing that are open to members of the public?

https://www.reddit.com/r/deeplearning/ seems to be mostly people that just took a Udemy / Coursera course, so it's usually about re-using existing models and almost no talk about research.

> BTW, are you aware of any discussion groups for AI image processing that are open to members of the public?

r/machinelearning is the most suitable subreddit for this, since it is actively frequented by researchers and the quality of discussion is much higher than in r/deeplearning

Thanks :)

I like the choice of user name

What about findings w. R. T. Online learning? I find continuous learning quality of algorithms to be a topic which often seems to be more of a side-concern, although it carries a lot of relevance in applied settings.

I believe that to be a red herring. Their approach cannot learn any features that provide a lower-dimensional approximation of the input data. As the result, there is no intermediate representation which could change and thereby negatively affect previously learned classifiers.

But if I train 10 independent traditional networks, I also won't have newly learned data affect old performance. So in effect they give up the possibility to do transfer learning in exchange for avoiding the disadvantages of transfer learning. But that's a bad tradeoff.

With their approach you always train from scratch, which brings with it the need for huge training data sets.

So I can train a bird classifier on the traditional architecture with 500 labeled images and a pretrained resnet. Our I use a million bird images and this approach.

1) Indeed, GLNs don't learn features... but I would claim they do learn some notion of an intermediate representation, it's just different from the DL mainstream -- in particular its closely related to the inverse Radon transform in medical imaging.

2) Inputs which are similar in terms of cosine similarity will map to similar (data dependent) products of weight matrices, and thus behave similarly, which of course can affect performance in both good and bad ways. With the results we show on permuted MNIST, its well... just not particularly likely that they will interfere. This is a good thing -- why should completely different data distributions interfere with one another? The point is the method is resiliant to catastrophic forgetting when the cosine similarity between data items from different tasks is small. This highlights the different kind of inductive bias a halfspace gated GLN has compared to a Deep ReLu network.

3) Re bird example, that's slightly unfair. I am sure one could easily make use of the pre-trained resnet to provide informative features to a GLN -- it's early days for this method, hybrid systems haven't been investigated, so I don't know whether it would work better than current SOTA methods for image classification. But I would be pretty confident that some simple combination would work better than chopping the head off a pretrained network and fitting an SVM on top. This is all speculation on my part though. :)

2) The problem that I would expect with a hybrid method is that conv features are usually trained to be redundant with dropout, so they should be highly correlated with each other and, thus, have a high cosine similarity.

3) I agree that my argument is scientifically unfair. I was trying to argue from the perspective of a prospective user. My customers tend to have a budget limit of how much their classifier is allowed to cost. Training from scratch would be too expensive. But a chopped reset with some conv layers will work OK and be cheap enough.

So for me the user, the ecosystem around your architecture and the availability of pretrained models might make the critical difference on whether I'll use it or not.

Good point as well - sometimes its not about dimensionality reduction but more about persistent representation, having this geared towards highly non-stationary environments is nice thing to have.

Still, there are a lot of domains where transfer learning is no the most applicable setting - I'm thinking of highly noisy and non-stationary setting such as finance. In some of these domains, especially time series, lack of data is often not the issue, e.g. high frequency datasets.

Having models constantly re-train as the default setting is essentially what a rolling regression would do - having a rolling regression that doesn't catastrophically forget would be quite valuable.

what stops you from using a few convolution layers and then use the GLN in the last layer? You achieve the best of both worlds if your theory is true.

That would work, but it would likely re-introduce the catastrophic forgetting problem, because now the GLN is dependent on an intermediate representation determined by conv layers.