{"body":"10:37 PM I am pretty sure the implementation of transposed convolution layer has multiple interrelated mistakes\n10:37 PM Also, I think one thing can be changed in convolutional layer as well\n10:37 PM agreed, I am disappointed by the spam...\n10:38 PM rcurtin: Is it happening to all freenode channels?\n10:38 PM most that I am in, yes\n10:38 PM I wonder if it is happening in #gsoc, I'll join there\n10:38 PM Atharva: If you can provide a simple test case with the expected output, I'll take a look, or perhaps you already fixed it?\n10:39 PM zoq: I can fix it, I know how. I just thought I will discuss with you first.\n10:39 PM Atharva: What's your idea with the conv layer?\n10:39 PM Atharva: Oh, nice, yeah sure.\n10:39 PM I think I will start with the transposed convolution, because with conv it's just performance improvemnent\n10:40 PM So, the output size of transposed conv is being calculated incorrectly\n10:40 PM according to this paper https://arxiv.org/pdf/1603.07285.pdf\n10:40 PM Atharva: Okay, I think any improvment would have a huge effect on the overall model performance.\n10:40 PM Yes\n10:41 PM Sorry for being slow, I am a little confused what to start with, there are a lot of things\n10:41 PM I will try my best to explain it all\n10:42 PM I think Shikhar opened an issue regarding the otuput formula, let's see if I can find it.\n10:43 PM zoq: That's great, but the problem is not just with the formula\n10:43 PM https://github.com/vdumoulin/conv_arithmetic/issues/18, no response so far\n10:44 PM The forward function of Transposed Conv is flipping the filter, that's not needed in this case\n10:44 PM It performs full convolution irrespective of stride and padding, which is incorrect\n10:46 PM This brings us to the stride, it's true that the effective final stride in transposed conv layer is always 1, but the operation does depend on the stride of the equivalent conv layer\n10:46 PM for example 4.5 and 4.6 of the above paper\n10:49 PM I see the issue with the full convolution and the stride parameter, but I'm not sure about not flipping the kernel.\n10:49 PM I think for trans conv layer, the stride parameter should take s instead of s`. s` is always 1, but we need s(stride of corresponsing conv operation of the given trans conv layer)\n10:49 PM We can flip the kernel in the backwars function instead of forward\n10:50 PM it seems more apt\n10:50 PM It won't matter mathematically I think\n10:50 PM yeah, that's the same\n10:52 PM Also, we are never inserting zeros between input units when the stride is not 1, section 4.5 and 4.6 of that paper\n10:52 PM What we instead do is always perform full convolution operation which is incorrect and extremely inefficient.\n10:53 PM For example, in my encoder network, I have a conv layer that goes from 64x64 to 32x32 with s = 2, p = 2, k = 5\n10:54 PM I think, inserting zeros is a result of the incorrect dimension, which I think isn#t effecting the output\n10:54 PM But, for the transposed equivalent in the decoder, I am forced to use a kernel size of 33 to go from 32x32 to 64x64(a full convolution)\n10:54 PM I see\n10:55 PM As mentioned in the paper, we are always doing 4.3 which they have said to be extremely inefficient \n10:56 PM The transposed conv layer took 30 seconds on mylaptop while the conv layer took about 0.5\n10:56 PM We need to correct the output size formula in transposed conv\n10:56 PM yeah, especially if you use such a huge kernel size\n10:56 PM agreed\n10:57 PM Yes, sadly that's the only option right now\n10:57 PM we also need to take the stride of the equivalent conv layer instead of actual stride of transposed conv(which is always 1)\n10:57 PM right\n10:58 PM We need to insert zeros between the input units when it's > 1\n10:58 PM Similarly, the backward function is also wrong\n10:58 PM It always performs a valid convolution with no padding on the error matrix even when it's needed\n10:58 PM yeah, same issues\n10:59 PM I think, it will be better to use valid convolution for both forward and backward in transposed conv and take care of padding and zeros in between manually\n11:00 PM that would also speed things up\n11:00 PM Yes\n11:01 PM Also, a minor thing, we can use the correct typenames `ForwardConvolutionRule` and `BackwardConvolutionRule` for the corresponding functions :)\n11:02 PM :)\n11:02 PM about conv layer now\n11:02 PM any corrections are much appreciated\n11:03 PM The backward function of conv layer performs a full convolution irrespective of the output(input) size it needs to output which leads to too many unnecessary operations.\n11:04 PM for example, let's say a conv layer gets input 5x5 with k 4x4 and padding 2, s =1\n11:05 PM it goes to 6x6\n11:06 PM in the backward function, what happens is, it pads the 6x6 input error to 12x12 first, then takes it to 9x9 with k(inverted) 4x4 and then only used the centre 5x5 of it\n11:06 PM we should instead just pad 6x6 to 8x8 and directly take it to 5x5\n11:10 PM interested to see the performance improvement, i think this would mostly effect bigger kernels\n11:10 PM Yes, should I change the conv part or just the transposed conv?\n11:11 PM I am not sure how better the performance will be in conv layer after that change, it was just somthing I noticed\n11:12 PM it should be faster, so if you like to take a look into it, I'm happy to merge this in\n11:13 PM zoq: Great! I should open a PR tomorrow if I don't run into any issues\n11:13 PM wow, that's fast\n11:13 PM thanks!\n11:14 PM zoq: Happy to help, I had to clear a lot of concepts to solve this, it was fun!\n11:14 PM I think we should discuss some implementation details for transposed conv\n11:14 PM 1) we take stride of equivalent conv operation\n11:14 PM what I really like is that this will effect all sorts of models including the rl code\n11:14 PM 2) change the output formula\n11:15 PM zoq: That's great! even the transposed conv?\n11:16 PM Right now, only the GAN and VAE code will use the transposed conv operation, but who knows.\n11:16 PM Yeah, right\n11:16 PM so I will continue\n11:17 PM 3) we need to add zeros between input units if s > 1, this we will do in the forward function and not in the naive conv class, is that okay?\n11:17 PM so, we will just perform valid conv operation after taking care of the zeros\n11:18 PM about 3) fine with me, that way we don't have to touch the naive conv code\n11:18 PM Yes\n11:19 PM 4) same thing with backward, we don't touch naive conv code\n11:19 PM right\n11:19 PM We will need to take care of the case when s > 1, because we only have to take alternate points from the output\n11:21 PM hm, in this case it would be nice to modify the conv rules, don't you think?\n11:21 PM Oh, so that we can take care of it when pointers in the valid conv function?\n11:21 PM Yeah, I think it will be more efficient\n11:23 PM in this case we would have to modify all conv rules, but we could start with the naive rule\n11:23 PM oh, okay. I don't think I know how the other conv rules work, can you suggest me something to read on it\n11:23 PM but I agree it should be faster if we do it inside the rule class\n11:24 PM the other one is based on fft\n11:25 PM don't think there is an easy way to skip the input, as we could do for the naive rule\n11:26 PM let's focus on the naive rule, I think it will outperform the fft rule afterwards (for small kernels)\n11:26 PM Oh, is it okay if we do it after gsoc? \n11:27 PM so, we could remove the code\n11:27 PM Yeah, I think\n11:27 PM of course\n11:28 PM 5) for conv layer backward function, for that change, we would need to manually add padding and use valid conv instead of full conv\n11:29 PM sounds reasonable\n11:30 PM Btw. everyone should think about the final report and put some work into it.\n11:31 PM Yes, will surely do\n11:32 PM I was thinking I will create a repo on my account and explain what I did over the summers with links to the PR and some results to show\n11:35 PM Yeah, if you like you can also write the report in the form of a blog post, something like:\n11:35 PM - http://www.mlpack.org/gsocblog/implementation-of-tree-types-summary.html\n11:35 PM - http://www.mlpack.org/gsocblog/deep-reinforcement-learning-methods-summary.html\n11:35 PM - http://www.mlpack.org/gsocblog/summary-of-lsh-changes-for-gsoc-2016.html\n11:36 PM but that's up to you. As for me the final report is somewhat of a living document, which can be updated even after GSoC has ended, like if we change/merge something afterwards I think the report should reflect that.\n11:41 PM Also, I think it's important that the report is visible, so if anyone is interested in what you did over the summer, there is an easy way to find out (the GSoC page will link to the final report).\n","name":"","extension":"txt","url":"https://www.irccloud.com/pastebin/bEq3kpwu","modified":1533757066,"id":"bEq3kpwu","size":9528,"lines":107,"own_paste":false,"theme":"","date":1533757066}